On-Call Without Burning Out a Four-Person Team

The SRE books were written by companies that can run follow-the-sun rotations across three continents. I run a four-person consultancy in Cyprus that holds operational responsibility for several client platforms. If we ran on-call the Google way, we'd have a rotation of one week in four, every incident escalating to the same two senior people, and a team that quits within a year.

We've had to work out what sustainable incident response looks like at our size. It's been running for over a year now without anyone burning out, so here's the system, including the parts we got wrong first.

Severity levels that mean something

Every incident framework has severity levels. Most of them are decorative, because nobody agrees in advance what they trigger. Ours are defined entirely by response, not by adjectives:

The single most important rule: SEV3 cannot page. Early on, a client's staging environment filling its disk paged our engineer at 2 a.m. twice in one week. Staging. Twice. That's not incident response, that's a tax on sleep, and it's how you teach people to ignore pages. Everything that is not a SEV1, or a SEV2 during the day, goes to a queue that a human reads at a humane hour.

The second rule: severity is set by the responder at acknowledgement time and can be downgraded without ceremony. Nobody needs permission to say "this alert is wrong" at 3 a.m. and go back to bed, as long as it's reviewed in the morning.

The alert budget: every page must be actionable

We treat pages like a budget because attention, like money, runs out. The standard is blunt: a page must require a human decision or action, now. If the correct response to an alert is "watch it", it's not a page. If it self-heals, it's not a page. If it fires more than a couple of times a month and the fix is the same every time, it's not a page, it's an automation ticket wearing a costume.

Every Monday we spend fifteen minutes reviewing every page from the previous week, across all clients, and each one gets exactly one of three verdicts: it was actionable (fine), the threshold or alert gets changed (most common), or the underlying thing gets fixed so it can't fire again. There is no fourth option. "It's fine, that one always flaps" is banned as a sentence.

When we inherit a client's monitoring, the first engagement week is usually deleting alerts. One client came to us with over 200 alert rules and a team that had push notifications muted, which is the monitoring equivalent of removing the batteries from the smoke detector. We cut it to around 30 rules that page and the rest demoted to dashboards. Detected incidents went up, because the pages that remained were believed.

Runbooks are the price of going on rotation

With four people, everyone is on call for systems they didn't build. The rule that makes this survivable: no service enters the rotation without a runbook, and no alert pages without a link to one.

The runbook standard is deliberately modest, one page, answering: what does this alert mean, what do I check first, what are the two or three most likely causes and their fixes, what's safe to restart, and who do I escalate to (including on the client's side) if none of that works. Written for a competent engineer who has never touched this system and is 40% awake.

The enforcement mechanism is cultural, not procedural: if you get paged for something without a runbook, the person who built that thing owes you the runbook by end of week, and the review of the incident says so. It took about three months of consistency for this to become self-sustaining. Now the team writes runbooks unprompted, because everyone has been on the receiving end of a missing one.

Postmortems: blameless, but not toothless

Every SEV1 and any interesting SEV2 gets a postmortem within a week. Blameless is table stakes, with four people, a culture of blame would end the company in a quarter. But blameless is often misread as consequence-free, where the document gets written, everyone nods, and nothing changes.

Ours end with a maximum of three action items, each with a name and a date, and they go into the same sprint board as client work, not a separate "someday" list. Three, maximum, because a postmortem with eleven action items is a postmortem with zero action items. We also track one meta-metric: has this class of incident happened before? A repeat incident with an unfinished action item from last time is the closest thing we have to a cardinal sin, not because someone failed, but because the system failed twice and we ignored what it told us the first time.

Negotiating SLAs you can actually honour

The hardest part of small-team on-call is the conversation before the contract. Clients ask for 24/7 with 15-minute response because that's what the enterprise template says. A four-person team can technically sign that. It cannot honour it for long without breaking people.

What we do instead is price honesty into the tiers: business-hours response is the default and is reasonably priced; genuine 24/7 SEV1 coverage exists but is priced to reflect what it costs in human terms, roughly triple. Then we walk through the client's actual incident history with them. Almost every time, the data shows their real SEV1 rate is a handful of events per year, and paying triple to shave 45 minutes off each is a decision they make rationally, usually declining. The clients who accept the higher tier genuinely need it, and the price funds making it sustainable. The one thing we refuse outright is aggressive SLAs at business-hours prices. A signed SLA you can't honour isn't revenue, it's a scheduled dispute.

Tooling on a budget

You need surprisingly little: alert routing with escalation and overrides (the free/cheap tiers of the mainstream pagers are fine at our size), Prometheus and Grafana or the client's cloud-native equivalent, a chat channel per client with a bot posting alerts, and a shared doc template for postmortems. Total spend is double digits per month. Every euro we've considered spending beyond that would have bought convenience, not reliability. The expensive part of on-call was never the tools, it's the human attention, and that's exactly the resource everything above is designed to protect.

The uncomfortable summary: small-team on-call works only if you're ruthless about what deserves a human's night. Everything else, the budgets, the runbooks, the SLA pricing, is scaffolding around that one idea.