5 useful things you'll learn in my new post-training textbook (shipping now!)
Nathan Lambert's new RLHF and post-training LLM textbook covers PPO, GRPO, GSPO, CISPO and related techniques, freely available online.
Nathan Lambert's book 'Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs' is now shipping from Manning. It covers policy-gradient algorithms including PPO, GRPO, GSPO, CISPO, and RLOO, plus loss aggregation, truncated importance sampling, asynchronous RL systems, and post-training topics like rejection sampling, outcome reward models, and on-policy distillation. The book is freely available online with a 12-hour course, codebase, and exercises.
- Textbook covers RLHF and LLM post-training from foundations to systems
- Includes derivations and intuitions for PPO, GRPO, GSPO, CISPO
- Freely available online with 12-hour course and codebase
- Targets readers with CS background, not pure beginners
From the original article
7 min read at interconnects.ai
After a few long years of finding time to document my lessons from training open models, my post-training book is done!
By Nathan Lambert · interconnects.ai
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.