2026 / Free online book · Field Manuals
Observability for AI Systems
Knowing why the model said that
Access
Free
Chapters
6
Read time
36 min
Alpesh Nakrani
CRO at Devlyn · former CTO & COO
Traditional observability assumes determinism. AI systems are probabilistic, so the question shifts from where it broke to why it drifted. This manual builds the tracing, prompt logging, and replay you need to debug a system whose behavior changes with every model version.
A stack trace tells you where code broke. What tells you why a model drifted? Tracing, logging, and replay for probabilistic systems.
This edition is free to read onsite. Each chapter has its own URL, so readers can bookmark, share, and return to the exact section they need.
Table of contents
01 The Production Problem Why the work fails after the demo and what must be made explicit first. 6 min 02 The Reference System The smallest system design that can be owned, tested, and improved. 6 min 03 Measurement That Changes Decisions How to measure quality, cost, risk, and user impact without vanity metrics. 6 min 04 Failure Modes and Recovery Where the system breaks, what early signals matter, and how to recover. 6 min 05 Operating Cadence Ownership, review cycles, release gates, and the rituals that keep quality honest. 6 min 06 The First Ninety Days A practical rollout plan from pilot to production without losing control. 6 min