Teach It to Chat
Base model → assistant: SFT, LoRA, DPO on one consumer GPU.
This quest is reachable without the pretraining capstone, on purpose. Post-training starts from open base models, and it's where most of the jobs are. TRL v1.0 is the stack, and its 2026 shape is itself a lesson: SFT, DPO, GRPO, and distillation stable, PPO demoted to experimental. “LoRA Without Regret” is the citation that changed default practice — adapters on all layers including MLPs, at roughly 10× the full-FT learning rate, match full fine-tuning for post-training workloads. That result is what makes one consumer GPU a legitimate post-training rig.
Read LIMA critically. Its thousand-example minimalism is true for style and format, not for capabilities; reasoning SFT uses six-figure trace counts. DPO survives as the preference stage in every serious open recipe (Tulu 3, OLMo 3) while RL owns capabilities, and that division of labor is a favorite interview probe. The chat-template rigor from the serving path applies here in reverse: now you're the one whose template bugs everyone else inherits.
SmolLM-class base, a real instruct dataset, the chat template applied and hand-verified (render one example yourself and diff it). Compare before/after generations.
All layers including MLP, ~10× LR, modest rank — the Thinking Machines recipe. QLoRA (arxiv.org/abs/2305.14314) puts 7B-class fine-tuning inside 8 GB.
“Quality over quantity” — now understood as true for style/format, false for capabilities. Knowing the 2026 read on it beats knowing the abstract.
TRL DPOTrainer on your SFT checkpoint. Understand the implicit-reward derivation well enough to whiteboard it — it gets asked.
The harness trains its own base model, then hands it to you frozen with a budget of “language B” — 8 tokens the base has never seen. Return a rank-≤8 LoRA (the harness applies it to ITS base): language B must reach the calibrated band while language A regresses ≤0.1. The two lessons are rehearsal and learning-rate discipline — skip either and watch A collapse.
build+200 XPauto-verifiedTokenizer → pretrain → SFT → chat UI, one repo, small enough to finish. The integrative build: every stage you've now studied, touched in sequence.