TL;DR
- Our AI code reviewer looked 99% effective. Searched for misses, it caught at most 47%.
- Model errors correlate, so a stronger supervisor can share more blind spots with the agent.
- Score a second checker on the first one's misses, never on how often it agrees.
- Gate on the lower bound: 16 of 34 catches spans roughly 32% to 63%.
- The management artifact for agents is a measured escape rate, not a dashboard.
Everyone agrees software should supervise the agents
The consulting consensus on AI agents has settled into one sentence: managers get wider spans, and software does the supervising. McKinsey partners describe AI as giving leaders the capacity to manage bigger scopes and flatten the org. IBM Consulting's Mohamad Ali said it most plainly: "There'll be systems to manage these things."
We agree with Ali. But look at what IBM showed when asked to demonstrate it: a real-time dashboard that a human checks every hour. That is visibility. It is not management.
A system that manages agents has to answer one question a dashboard can't: how often does the thing checking the agent miss? Almost nobody measures it. When we measured our own, the answer was half what we'd been reporting.
The supervisor is also a model
"Systems manage agents" quietly assumes the supervising system fails independently of the agents it supervises. In practice the supervisor is usually another language model, and model errors are correlated.
An ICML 2025 study of more than 350 models found that when two models both get a question wrong, they pick the same wrong answer about 60% of the time on one leaderboard. Larger, more accurate models were more correlated, even across vendors. Used as judges, models inflated the scores of weaker models, especially from their own family.
That has an uncomfortable implication. Upgrading your supervisor to a stronger model can make it worse at catching the agent's errors, because the two now share more blind spots. The errors a correlated checker misses are exactly the plausible-looking ones, which are the ones that reach customers.
So the management question isn't "is there a checker?" It's "what is the checker's catch rate on the errors this agent actually makes?" That number has to be measured, and measuring it honestly is harder than it looks.
Our checker catches half of what we thought
Our pull requests are reviewed by a commercial AI code reviewer. Across three months of merged PRs, its record looked excellent. On the 106 PRs where nobody went looking for misses, it caught 231 of the 233 defects we logged: 99%.
That number was wrong, and the reason is structural. We only logged a defect as missed when someone happened to notice it. A catch rate whose denominator only includes the errors you stumbled on will always look good.
So we built a golden set: seven PRs where someone independently searched for everything the reviewer should have flagged. On those, the reviewer caught 16 defects and missed 18. Its honest catch rate is at most about 47%, not 99%. Models helped find some of those misses, so the true rate may be lower.
| Measurement | PRs | Catch rate |
|---|---|---|
| Naive (misses counted only when noticed) | 106 | ~99% |
| Audited (misses searched for independently) | 7 | ~47% (upper bound) |
Two decisions followed. First, the naive history was not written into our supervision stats, because an automated merge gate reads that number and would have acted on a biased one. Catch rates are now tracked forward only, seeded with the audited set. Second, the gate won't act at all until the sample is large enough to mean something.
The mistake we almost made
A checker that misses half its defects needs its own adversary. We set up an open-source reviewer as a non-blocking second opinion on the commercial one, then ran a bakeoff of three models from two vendors on the audited set.
Each model independently caught 8 of the 16 real defects the commercial reviewer had flagged. Each caught a different 8, and together they caught 11. We nearly picked a default on cost per finding.
Then we noticed what we'd scored. "Caught 8 of the commercial reviewer's findings" measures agreement with the checker we were trying to check. For an adversary, agreement is the least valuable thing it can produce. The only score that matters is its catch rate on what the first reviewer missed, and just one of the three models had been scored on that.
The same trap hides in a tempting heuristic: treat findings that every model flags as the highest-signal ones. For precision, that's true. But the five defects all three models agreed on are, by definition, the correlated set. The value of a second checker lives in the singletons: defects only one model found, and above all the ones the first reviewer missed.
The lesson generalizes. Score an adversary on the primary's misses, never on its agreement with the primary. Otherwise you select for the checker most like the thing it checks, which is the opposite of decorrelation.
Rules for measuring a checker
This is what we now hold our own supervision to. Some of it is already enforced; some is still being wired in.
- Measure on organic errors. Catch rate comes from errors the agent actually made, found by an independent search. Seeded or synthetic bugs are easier to catch than the plausible ones real agents produce.
- Know who searched for the misses. If a model helped find them, the denominator inherits that model's blind spots. The honest number becomes an upper bound, not a value.
- Gate on the lower bound. Sixteen catches out of 34 defects is a point estimate of 47%, but the 95% interval runs from roughly 32% to 63%, wider once defects cluster within PRs. A gate that reads the point estimate is acting on false precision.
- Stratify by the agent's confidence. The errors that matter most are the agent's confident ones. We've seen a confidence score in our own stack that was anti-calibrated: its top buckets needed fixes more often than its bottom ones.
- Score adversaries on misses, then rotate them. A fixed checker gets learned; agents tuned against it learn to pass it rather than to be right.
- Treat the composite as a trust score, not a probability. Discounting confidence by catch rate is a sound heuristic. Read as a probability, it assumes the independence the research says you don't have.
What we still haven't measured
Code review is the only layer of our supervision with meaningful data, and even there the sample is eight PRs. We supervise agents at six layers, from inside a single agent's run up to deployment and strategy. The other layers have one recorded observation or none, and none has had a drift check.
Human gates are no better. No approve button in our fleet has had its catch rate measured on injected errors. By our own rule, an unmeasured gate is decoration, and that includes the ones staffed by people.
Last month we wrote that every number FRIDAY, our operational memory system, produces is discounted by an adversary with a measured catch rate. The formula is real and it runs. The measured basis behind it is still thin, and we'd rather say so here than have a client find it first.
The honest number is the product
If software is going to manage agents, the thing it manages is a defect distribution, not a workforce. The management artifact isn't a dashboard. It's a measured escape rate per task type, with an audit trail showing how the misses were found.
Any catch rate computed without searching for misses will look like our 99%, and it's the kind of number most AI tooling reports. Our 47% is less flattering and far more useful. It tells us where the next adversary has to go, and it's a number a buyer can hold us to.
Dashboards show you the agents. Measurement tells you whether to trust them.
Aiterated builds AI systems with adversarial, decorrelated checking as a first-class requirement. Related: No learner without a decorrelated checker · Human-in-the-loop is not a safeguard · The approve button is not a safeguard · Why your AI should disagree with itself.