Skip to main content

Runbooks

One page per pageable alert. When an alert fires, the on-call opens the linked runbook and works through it. Runbooks are code — reviewed on PRs, kept current with the system they respond to.

Alert catalog and SLO framing: architecture/observability.

Template​

Every runbook uses the same shape. Copy this when adding a new one.

## <alert-slug>

### What fired
<the alert condition — e.g. "API 5xx rate > 5% over a rolling 5-min window">

### Severity
Page (SEV-1) | Ticket (SEV-2) | Ticket (SEV-3)

### First checks (in order)
1. Open <dashboard-link>
2. `wrangler tail --format pretty` — look for stack traces
3. Check <upstream-vendor-status-page>
4. Confirm the alert isn't a false positive (metric labels, time window)

### Common causes and fixes
- **Cause A** — symptom → what to do
- **Cause B** — symptom → what to do
- **Cause C** — symptom → what to do

### If none of the above
1. Notify #incidents in Slack with alert link + first-check findings
2. Consider rollback: `wrangler rollback` if last deploy was in the causal window
3. Escalate to <next person or role>

### Postmortem trigger
Open a postmortem if any of:
- User impact > <threshold>
- Duration > <threshold>
- Fix required a rollback
- Root cause is unclear after resolution

Runbooks (to be authored when alerts are wired)​

Status: none live yet. Each entry below is a placeholder for a runbook that lands in the same PR that wires its alert. Grouped by the tier from observability § alerts.

Page (SEV-1)​

  • api-5xx-spike — API 5xx rate > 5% over 5 min
  • infinia-outage — Infinia error rate > 50% for > 5 min on any endpoint
  • webhook-backlog — > 100 unprocessed webhook_events older than 60s
  • payout-failure-spike — payout failure rate > 5% in a 10-min window
  • neon-outage — Neon connection failures > 30s continuous
  • jwks-outage — Apple / Google JWKS fetch failure rate > 20% for > 5 min

Ticket (SEV-2 / SEV-3)​

  • slo-burn-fast — burning error budget at 14× over any 1h window
  • slo-burn-slow — burning error budget at 6× over any 6h window
  • recon-drift — nightly reconciliation reports drift
  • kyc-decline-spike — decline rate ≥ 2× rolling 7-day average
  • retention-cron-fail — nightly retention cron errored
  • brl-residual-high — any user's BRL account > R$100 outstanding > 24h

When adding a new runbook​

  1. Land it in the same PR that wires the alert. No alert without a runbook.
  2. Use the template above.
  3. Cross-reference it from the alert row in observability § alerts.
  4. If the runbook grows large (multiple pages of prose), promote it to its own file under apps/docs/docs/ops/runbooks/<slug>.md and update the sidebar.

Postmortems​

Not runbooks, but adjacent. Every SEV-1 gets a postmortem within 5 business days. Template + index live under apps/docs/docs/ops/postmortems/ (to be created when the first one lands). Postmortems are blameless and public within the team.