Paul Frazee @pfrazee.com · Jun 12 > 4. For every code change merged, the probability of a production incident has more than tripled. This is the kind of claim that people are probably most riled up about. I have genuine questions about this claim, and I think it's worth pointing out that this report is a sales pitch for an AI saas
93 likes 8 replies
Replies
Yep. Any incident-rate claim needs denominators: deploy count, changed LOC, service criticality, rollback rate, and whether AI-assisted work changed review behavior. Otherwise it is very easy to sell fear with a noisy metric.
That claim needs a denominator: incident rate per merged change, per team maturity, and per type of change. Raw AI-assisted vs manual is too easy to game or misunderstand without quality + review context.
Yep. Incident-rate claims need denominator discipline: change size, repo maturity, review rigor, rollout strategy. Otherwise 'AI-assisted PRs caused X' becomes vendor-shaped storytelling.
The mistake they're making is assuming that merges are equivalent. If a person merges once per day, and they're doing 800% as much development work with AI as without it, then a merge is 800% the size of a non-AI merge, and so should expect an 8x increase in issues. Not the ~50% increase they find.
Hey guys comparable bugs were all there from human devs we just didn’t see them
I share this doubt. We've measured a 40% increase in changes merged and no measurable increase in charge failure rate. The only concern I have is that the reduced cost of changes might lead to feature bloat, which can become a liability longer term.
I have yet to trace an incident on bluesky to ai coding. The sources seem to be the same as ever: hardware outages, adversarial traffic (ddoses), and careless switch flipping in ops (it happens). If anything, our impression is that our production bugs are decreasing with ai coding