What is a runbook?
A runbook is a written procedure that tells an operator exactly how to carry out a specific operational task or respond to a known type of incident. It lists the trigger, the steps, the expected result of each step and what to do if a step fails. Runbooks cover routine work, such as rotating certificates or adding storage to a database, and recovery work, such as failing over a service or clearing a full disk. The aim is repeatability. A runbook written and tested in advance lets an on-call engineer who did not build the system resolve the issue correctly at 3 a.m.
What a good runbook contains
Good runbooks are short, specific and tested. Each one should be usable by someone who knows the platform in general but has never seen this particular failure before. Write steps as commands or clicks, never as goals such as "check the database". Use the same template for every runbook so engineers know where to look for rollback and escalation details when they are under pressure.
- Scope and trigger: which alert or request starts it and which systems it covers
- Prerequisites: access, tools and approvals needed before starting
- Steps: exact commands or actions, each with its expected output
- Verification: how to confirm the task worked
- Rollback and escalation: how to undo changes and whom to call if a step fails
- Owner and review date: who keeps it current and when it was last tested
Runbook vs playbook vs SOP
The terms overlap and organizations use them differently, but a common split works well. A runbook covers one task or one failure with concrete steps. A playbook covers a broader scenario, such as a major outage or a security incident, and defines roles, decisions and communication, often calling several runbooks along the way. A standard operating procedure (SOP) is a formal, approved description of how routine work must be done, often written to satisfy audit or compliance requirements. A single SOP for server patching might point to separate runbooks for Linux and Windows hosts.
Worked example: a disk space runbook
Consider an alert that fires when the /var file system on a Linux application server passes 90 percent. The runbook first asks the engineer to confirm usage with df -h, then find the largest directories with du. If application logs under /var/log/app exceed 10 GB, it compresses logs older than 7 days, moves them to an archive share and rechecks usage against a target below 75 percent. If usage is still above 90 percent, the runbook stops and escalates to the application owner, because the likely cause is a logging fault that needs a configuration or code fix. Each run is recorded on the incident ticket.
Test each runbook in a nonproduction environment at least once per review cycle and record the result on the runbook itself.
From manual runbook to automation
Runbooks usually mature in stages. A new procedure starts as a document written after an incident. As it is used, steps are refined and commands become ready to copy and paste. Next, the steps are packaged as a script that an engineer launches with one action after checking the situation. Finally, low-risk runbooks can run automatically when a matching alert fires, with the result posted to the ticket. Runbook automation removes skipped steps and typing errors, and it also removes human judgment. Keep a human approval step for actions that restart production services, change data or touch many devices at once.
Keeping runbooks current
Runbooks decay when systems change and documents do not. Link each runbook to the configuration items and alerts it covers, so a change to those CIs prompts a review. Review a runbook after every incident in which it was used, and retire runbooks whose systems no longer exist. Store them where on-call engineers already work, such as the incident tool or a wiki page linked from the alert notification, since a runbook that takes five minutes to find is often skipped.
How CloudSino approaches runbooks
CloudSino's iDCOS platform turns repetitive operations into automated workflows. It manages inspections, script execution, batch operations, reports and common alert handling actions, and places these operations under permissions, audit and result feedback. Alerts can be associated with resources and configuration items, trigger events and assign tasks to responsible parties according to rules, with the handling process recorded. A knowledge base sits alongside the workflow. For hardware recovery steps, CloudSino DCOS provides remote power on, power off, restart and KVM-over-IP console access through server BMCs, which remain reachable when the operating system is unresponsive.
Frequently asked questions
What is runbook automation?
Runbook automation is the execution of runbook steps by software instead of a person typing them. It ranges from a script an engineer starts manually to fully automatic remediation triggered by an alert. Each automated run should be logged with its inputs, its outputs and the user or rule that started it.
How long should a runbook be?
As short as possible while still being complete. Most good runbooks fit on one screen or one printed page. If a procedure needs many branches or several teams, split it into smaller runbooks and use a playbook to coordinate them.
Who should write runbooks?
The engineers who build and run the system should write the first version, ideally right after they resolve an incident or complete the task. A second engineer who did not write it should test it, since that person is closer to the on-call reader who will use it under pressure.