Every platform team describes this as a tension. Almost none of them plot it. Deploy frequency, lead time, change failure rate, and MTTR turn it into a two-axis instrument panel — and a worked canary/rollback scenario shows exactly where an AI risk-gatekeeper earns its keep, and where it just borrows confidence it hasn't paid for.
"Move fast" and "don't break things" get talked about like a philosophy. They're not — they're four numbers. Deploy frequency and lead time for changes measure velocity. Change failure rate and MTTR measure confidence. These are the four DORA metrics, and the reason they matter here is that they turn a vague tradeoff into a panel you can actually read: two gauges climbing on one axis, two gauges climbing on the other, and a real question about whether they're climbing together or trading places.
The pitch for AI as a release-risk gatekeeper is that it lets both climb at once — ship faster because a canary judge is watching closer than a human ever could, and roll back faster because the decision doesn't wait for someone to notice a dashboard. That pitch is sometimes true. It's also the exact spot teams talk themselves into a false reading, so it's worth building the panel properly before trusting what it says.
Sample readings for a mid-size service, mid-migration to AI-assisted canary analysis. Two gauges belong to velocity, two to confidence — and the honest version of this post only exists if both pairs are shown side by side, not just the ones that flatter the migration.
Lower is better for change failure rate and MTTR — the needles read "good" toward the low end, which is why the dial fills small rather than large for those two. That's a deliberate inversion, and it's exactly the kind of detail that gets lost when a dashboard just shows four numbers with no shared frame.
The pitch earns its keep or loses it in exactly this window. A canary at 5%, a promotion to 25%, and a signal that crosses threshold four minutes later — recorded like a flight log, because that's the level of scrutiny a rollback decision deserves.
That 44-second number is the entire pitch in one line. A human on-call, paged and context-switching from something else, is not reverting a bad canary in 44 seconds — and every minute that gap stays open is a minute of the 25% cohort taking the hit. The gatekeeper didn't make the release safer by judging better than a person would. It made it safer by not needing to be paged.
Strip the phrase "AI as gatekeeper" down and there's a scoring function, two weighted deltas, and two thresholds. It's worth looking at directly, because the thresholds are where all the actual judgment lives — and they're set by a person, not discovered by a model.
interface CanarySignals { canaryErrorRate: number; baselineErrorRate: number; canaryP95: number; baselineP95: number; canaryCpu: number; baselineCpu: number; } function confidenceScore(signals: CanarySignals): number { const errorDelta = signals.canaryErrorRate - signals.baselineErrorRate; const latencyDelta = signals.canaryP95 - signals.baselineP95; const saturationDelta = signals.canaryCpu - signals.baselineCpu; return ( weightedRisk(errorDelta, ERROR_WEIGHT) + weightedRisk(latencyDelta, LATENCY_WEIGHT) + weightedRisk(saturationDelta, SATURATION_WEIGHT) ); } // promote only after N consecutive windows below PROMOTE_THRESHOLD // roll back immediately on a single window above ROLLBACK_THRESHOLD — // asymmetric on purpose: promotion earns trust slowly, rollback spends it fast
The failure mode isn't the gatekeeper making a bad call. It's a team that raises deploy frequency because the gatekeeper exists, without ever tightening the two numbers that actually back that decision up: rollback latency and MTTR. Velocity climbed because someone trusted the panel. Confidence didn't climb with it — it just went unmeasured for a few sprints, until a canary window's blast radius turned out to be bigger than 25% of traffic for 44 seconds.
The tell is in the thresholds, not the dashboard. If ROLLBACK_THRESHOLD hasn't been revisited since the gatekeeper shipped, and deploy frequency has doubled since then, the panel is showing two velocity gauges climbing on the strength of confidence gauges that were calibrated for a slower, smaller blast radius. That's not a hypothetical — it's the default trajectory of any team that treats "we have an AI gatekeeper" as a one-time upgrade instead of a dial that needs re-tuning every time volume changes underneath it.
Release velocity and release confidence aren't opposing forces — they're two axes on the same panel, and an AI risk-gatekeeper is a way of paying for velocity with faster decisions instead of slower ones. But it only works if MTTR and rollback latency are tracked as closely as deploy frequency is celebrated. A team that only watches the velocity gauges climb is reading half the panel, and the other half is exactly where the incident is waiting.