2026 / Free online book · Technical Deep Dives
Fine-Tune, or Don't
A Practical Decision Process for Customizing AI Models
Access
Free
Chapters
17
Read time
176 min
Alpesh Nakrani
CRO at Devlyn · former CTO & COO
Fine-tuning is reached for too early and dismissed too fast. This deep dive is a decision procedure: when prompting, retrieval, or routing solves it cheaper, and the narrow cases where changing the weights is genuinely the right tool.
Fine-tuning is the first instinct and usually the wrong one. A sober procedure for deciding when weights should change.
This edition is free to read onsite. Each chapter has its own URL, so readers can bookmark, share, and return to the exact section they need.
Table of contents
FM Front Matter: Fine-Tune, or Don't A Practical Decision Process for Customizing AI Models 5 min INT Introduction: The Wrong Operation, Performed Well A team I will call the support team, the details are composited from several real projects, but the shape is exact, had a problem that sounded like a fine-tuning problem. They ran tier-one customer support for a fast-moving SaaS product. 9 min 01 The Support Bot That Knew the Old Product > **Working claim:** Most "we should fine-tune" decisions are made before anyone has named what they are trying to change. 9 min 02 What Fine-Tuning Actually Changes > **Working claim:** A fine-tune does not "teach the model your data" in the way the phrase implies. It nudges a probability distribution toward the patterns your examples demonstrate. 9 min 03 Five False Diagnoses This chapter turns five false diagnoses into a concrete operating problem for the fine tuning or not book. 11 min 04 The Customization Menu and a Decision Tree > **Working claim:** Fine-tuning is one item on a menu of nine customization techniques, most of which are cheaper, faster, and more reversible. 8 min 05 Format, Behavior, and the Shape of a Repeated Task > **Working claim:** This is where fine-tuning shines. 9 min 06 House Style, Domain Phrasing, and Tool Discipline > **Working claim:** Three behaviors are worth training that teams often try to prompt forever: a genuine house style, fluent domain phrasing, and disciplined tool use. 8 min 07 Specializing Small Models and Distilling Down > **Working claim:** The most economically compelling fine-tune is not making a big model smarter, it is making a small model good enough. A large frontier model used as a generalist on a narrow, high-volume task is paying for capability it does not use. 9 min 08 Demonstrations, Corrections, and Preferences > **Working claim:** A fine-tuning dataset is the model update, compiled. 9 min 09 Labels, Disagreement, and Coverage This chapter turns labels, disagreement, and coverage into a concrete operating problem for the fine tuning or not book. 9 min 10 Contamination, Leakage, and the Splits That Save You > **Working claim:** The most common way a fine-tune lies to you is through the splits. If training and test data overlap, your eval reports a number that production will not honor, and you will ship a model you believe is good. 8 min 11 Synthetic Data: When It Helps, When It Poisons > **Working claim:** Synthetic training data, examples generated by a model rather than collected from reality, is the most powerful and most dangerous tool in the data toolkit. 9 min 12 SFT, LoRA, and QLoRA in Practical Terms > **Working claim:** You do not need the math to make the decision. 8 min 13 Preference Tuning, DPO, and Distillation > **Working claim:** When the target has no single right answer but a clear better-and-worse, demonstrations are the wrong tool and preference tuning is the right one. 9 min 14 What to Fine-Tune: Generators, Retrievers, and Routers > **Working claim:** "Fine-tune the model" almost always means "fine-tune the generator, " and that is frequently the lowest-impact place to spend a training run. 8 min 15 Baselines, Regression Walls, and the Release Gate > **Working claim:** A fine-tune you cannot evaluate against its alternatives is a fine-tune you cannot justify. 9 min 16 Versioning, Lineage, Drift, and Retirement > **Working claim:** Training a model is the beginning of the operational burden, not the end. A fine-tuned model is a thing you now version, reproduce, monitor, retrain, retire, and answer for, to auditors, to regulators, to the on-call engineer at 3 a. 9 min 17 Ten Playbooks for the Decision Meeting > **Working claim:** Everything in this book reduces to a decision a team makes in a room: *should we fine-tune this, and if so, how?* This chapter is ten such decisions, worked. 12 min A Appendix A: Back Matter Glossary, implementation checklist, and source register for the book. 9 min