Source: Dev.to | Jan 18, 2026 (UTC)
Incident Response Runbook Template for DevOps
Incidents are stressful when the team is improvising. A simple runbook reduces MTTR by making response repeatable, not heroic.
This post provides a ready to use incident response runbook template plus a practical Linux triage checklist you can run from any box.
What this runbook optimizes for
- Fast acknowledgement and clear ownership (Incident Commander + roles).
- Early impact assessment and severity assignment to avoid under/over‑reacting.
- Communication cadence and “known/unknown/next update” structure that builds trust.
- Evidence capture (commands + logs) to support post‑incident review.
Copy this into your internal wiki, README, Notion, or ops repo.
Triggers:
-
Monitoring alert / SLO breach
-
Customer report escalated
-
Internal detection (logs, latency spikes, error spikes)
-
Acknowledge page/alert in your paging system.
-
Create an incident channel: #inc-YYYYMMDD-service-shortdesc.
-
Assign Incident Commander (IC) and Comms Lead.
-
Start an incident document: timeline + links + decisions.
3. Assess severity (5–10 minutes)
Answer quickly:
– What’s impacted (service, region, feature)?
– How many users / revenue / compliance impact?
– Is impact ongoing and spreading?
Suggested severity:
– SEV1: Major outage / severe user impact; immediate coordination.
– SEV2: Partial outage / significant degradation; urgent but controlled.
– SEV3: Minor impact; can be handled async.
4. Stabilize first (10–30 minutes)
Goal: stop the bleeding before chasing root cause.
Typical mitigations:
-
Roll back the last deploy/config change.
-
Disable a feature flag.
-
Scale up/out temporarily.
-
Fail over if safe.
-
Rate-limit or block abusive traffic.
5. Triage checklist (host-level)
Run these to establish the baseline quickly (copy/paste friendly).
CPU
ps aux --sort=-%cpu | head -15
Enter fullscreen mode
Exit fullscreen mode
Залишити відповідь