Reliability is half the job we do. If something is already burning, tell us; this guide is for building the muscle that prevents the call.
The one-page rule
If a runbook needs scrolling to find the first action, it fails. Structure every runbook as one page:
- What is this? One line: the service, the owner, the page it should be on.
- How do I know it's this? The exact symptoms and dashboards that confirm it.
- What do I do first? The single highest-probability mitigation, step one.
- Who do I escalate to? Real names or on-call rotas, not "the platform team."
Write for the person at 3am
- Use one voice, commands with exact flags, and copy-paste blocks that are safe to run.
- Prefer five accurate steps over fifteen elegant ones. Accuracy of the path beats completeness.
- Put the dangerous do-not-do list near the top, not in a caveat at the bottom.
A template that works
## Service: (name) - owner: (team)
Symptoms: (dashboards / messages that confirm it's this service)
Confirm: (page URL or command that yes/no this is the failure)
Step 1 (most likely): (exact command + expected output)
Step 2: (if still failing)
Escalation: (name / rota)
Do NOT: (the one thing that makes it worse)
Review link: (post-incident template)The review that makes it real
A runbook decays the moment it's written. Two habits keep it alive:
- A runbook is a deliverable. Every pipeline, cluster, or migration ships with its runbooks, reviewed in the same PR as the code.
- Incidents write their own. After every incident, the fix summary becomes a new runbook entry or a correction to the one that failed you.
Six runbooks that match reality beat sixty that match an ideal world. Start there.