A year ago I wrote about vibe coding and the guardrails that keep you the architect when an agent writes your code. The stakes there were manageable: worst case, a bad PR gets reviewed and rejected. Since then, the agents have arrived in operations, and the stakes changed. A bad PR waits for review. A bad terraform apply does not.
We now use AI agents daily in our consulting work, including on systems that page us when they break. This post is the ops sequel to the guardrails piece: where agents actually earn their keep today, the permission model we insist on before any agent touches client infrastructure, and the failures that taught us the model.
Where agents actually help today
After a year of honest use across several client environments, the wins cluster in four places, and they share a property: the agent produces an artifact a human judges, rather than an action a human discovers.
Runbook drafting. The eternal problem with runbooks is that nobody writes them. It turns out an agent with read access to your Terraform, your alert rules, and your last three incidents drafts a genuinely decent runbook in minutes, and a human spends twenty minutes correcting it instead of three hours staring at a blank page. The correction step is where the value locks in; the draft is where the activation energy disappears. Our runbook coverage roughly tripled in six months, and the reason is embarrassingly simple: editing is easier than writing.
Log and alert triage. "Here are 4,000 log lines from the window around the incident, group them, find the anomaly, correlate with this deploy timeline." Agents are legitimately good at this, better than grep and faster than a tired human. The output is a hypothesis, and hypotheses are cheap to verify and safe to be wrong about.
Terraform review. We run an agent pass over every infrastructure PR before human review: security group changes, IAM policy diffs against least-privilege, deprecated arguments, drift between module versions across environments. It catches a different class of issue than humans do, humans miss the boring inconsistencies, agents miss the contextual "this is technically fine but strategically wrong". The combination is stronger than either. It has flagged real problems, including an IAM policy that quietly widened Resource: "*" in a refactor that looked cosmetic.
Incident summaries and comms. During an incident, someone has to write the client-facing updates and, later, the postmortem timeline. Feeding the agent the chat transcript and alert history and getting a clean draft timeline back saves the incident commander's attention for the actual incident. Every date, metric, and claim gets checked, see failure story number two, but checking is cheap.
Notice what's not on the list: remediation. We do not have agents restart services, roll back deploys, or "fix" anything autonomously. Not because it never works, but because of the asymmetry: the wins are minutes, the losses are outages.
The permission model
Everything above runs under a model we now apply uniformly, and which I'd summarise as: treat the agent like a smart intern on their first week, forever.
Read-only by default, and scoped. The agent gets a dedicated identity, never a human's credentials, with read access to logs, metrics, configs, and code for the systems in scope. Not admin-read on everything: an agent summarising app logs doesn't need to read the secrets manager, and "read-only" on a secrets store is not read-only in any meaningful sense. Least privilege applies to agents at least as strictly as to humans, because the agent will cheerfully use every permission it has.
Mutations are proposals. Any state change the agent wants must materialise as a reviewable artifact: a PR, a Terraform plan, a generated script with a diff. A human with the appropriate access reads it, and executes it through the normal pipeline. The agent never holds apply-capable credentials. This is the load-bearing wall of the whole model, and every convenience argument for weakening it, "but it's just restarting a pod", is how you end up with failure story number one.
Audit everything. The agent's identity means its every API call lands in CloudTrail or the equivalent, distinguishable from human activity. We log the prompts and outputs too. When something looks weird, "was this a human or the agent" needs a five-second answer, not a forensic investigation.
Kill switch and blast-radius caps. Revoking one identity's credentials must stop everything, instantly, and someone on the team must know they're allowed to pull it without asking. Rate limits and scoping keep even the read side bounded, an agent in a retry loop against your cloud API is a self-inflicted DDoS with a subscription fee.
Two failures that shaped the model
The helpful cleanup. Early on, in a client's staging environment, we ran an agent with broader permissions than it needed, because staging, right?, and asked it to investigate a disk-space alert. It correctly diagnosed old Docker images, and then, being helpful, pruned them. Including images that were the only remaining copies of two services whose CI had been broken for weeks, which nobody knew until the next deploy attempt. Staging was down for a day. The agent had done nothing a human wouldn't have done, that's the uncomfortable part, but a human would have paused at "why are these images unreferenced?", and the agent's job, as we'd framed it, was to fix the alert. Diagnosis and remediation have been separated ever since.
The confident timeline. An agent-drafted postmortem stated that a deploy at 14:32 caused the incident at 14:47. Clean narrative, plausible, and wrong, the deploy was to an unrelated service, and the actual trigger was a certificate expiry the summary never mentioned. We caught it in review, but only because the reviewing engineer had lived the incident. The lesson wasn't "agents hallucinate", everyone knows that. It was that narrative fluency masks factual gaps: a confident, well-structured document gets shallower review than messy human notes. We now require every timestamp in an agent-drafted postmortem to link to its source alert or log line.
About "AIOps"
The marketing around AIOps deserves a paragraph of cold water. Most of what's sold under that banner is anomaly detection with a new label, useful, occasionally, but the pitch has inflated to "self-healing autonomous operations", and I have yet to see that survive contact with a real production environment. The vendors demo remediation on the failure modes they trained on; production specialises in the ones nobody trained on.
The current honest state of the art is narrower and better: agents compress the toil around incidents, the reading, correlating, drafting, and summarising, so that human judgment gets applied faster and with better context. That's a real, large win. It is not autonomy, and teams buying "autonomous" today are mostly buying a demo and a liability question nobody can answer: when the agent breaks production, whose incident is it?
The guardrails conclusion from my vibe-coding post transfers almost verbatim: the agent accelerates, the human stays accountable. In ops the corollary bites harder, because the feedback loop is production itself. Give the agents the reading. Keep the writing, to your infrastructure, at least, behind a human's hands, on a human's credentials, in a pipeline you already trust. The teams getting real value from this today aren't the ones who automated the most. They're the ones who drew the line in the right place and actually held it.