Make the diff prove itself

The pull request looks fine. It compiles, the diff is small, the naming matches the codebase, and it arrived with tests. The tests are green.

This is the failure mode nobody had to worry about before, because badly written code used to look badly written. Developers already have a name for it: asked what frustrates them most about AI tooling, the top answer in Stack Overflow’s 2025 survey, at 66%, was “AI solutions that are almost right, but not quite.” Almost right is the hardest thing there is to catch by reading, because it was produced by a process optimising for the impression that reading gives you.

So look at the tests instead. In June 2026 a group of researchers took 86,156 test-file patches from 33,596 agent-authored pull requests across 2,807 repositories, covering five different coding agents. Their headline: 80.2% of those test patches contained weak or no explicit oracle signals. The largest single category was the one where no assertion pattern is present at all.

That is a preprint, and worth treating as one. But it names something you have already seen. The tests exercise the code. They do not check it. They go green because nothing in them can go red.

The move

Gate the change, not the codebase, and give the gate a budget in minutes.

Two halves, and both matter. Scoped to the diff, because a rule that runs against your whole codebase either takes an hour or gets weakened until it passes. Budgeted, because the alternative is a suite that grows until people start merging around it.

The budget number is older than any of this. The ten-minute build is one of the thirteen primary practices in Kent Beck’s Extreme Programming Explained, second edition, from 2004. Martin Fowler still endorses it: “For most projects, however, the XP guideline of a ten minute build is perfectly within reason.” Nothing about agents changed the arithmetic. It only raised the number of pull requests competing for those ten minutes.

Five gates that fit

Types, at maximum strictness. The cheapest deterministic check you own, and the only one on this list that runs against everything without costing you anything.

Mutation score, on changed lines only. This is the direct answer to a test that asserts nothing. Mutation testing deliberately breaks your code and demands that a test notice. A test with no assertions cannot notice, so it fails the gate that coverage happily passes. Scoping it to the diff is what makes it affordable: cargo-mutants takes --in-diff, which “tests only mutants that overlap with regions changed in the diff.”

Coverage delta, not an absolute threshold. Codecov splits this cleanly: its patch status “only measures lines adjusted in the pull request”, while the project status measures the whole thing. Gate on the first. An absolute project threshold punishes whoever touches a legacy file and tells you nothing about the change in front of you.

A contract check when the API surface moves. oasdiff breaking --fail-on ERR compares two OpenAPI documents and exits non-zero on a definite break, where the tool’s own definition of ERR is “definite breaking changes which should be avoided” as distinct from WARN, which it cannot confirm programmatically.

A screenshot check when a route moves. Playwright’s --only-changed runs “only test files that have been changed between ‘HEAD’ and ‘ref’”, and Chromatic’s TurboSnap narrows by dependency graph.

What it looks like

THE TEN MINUTES, SPENT ON THE DIFF

  types       tsc --noEmit                              whole repo, seconds
  mutation    cargo mutants --in-diff pr.diff           changed lines only
  coverage    codecov/patch                             lines the PR touched
  contract    oasdiff breaking --fail-on ERR \
                base/openapi.yaml head/openapi.yaml     only if the spec moved
  visual      playwright test --only-changed            routes the PR touched

  deliberately NOT here:
  the full mutation run, the full visual suite, the long fuzz.
  those are nightly. a nightly failure is a ticket, not a blocked merge.

The shape is the point. Every gate above answers a question about this change, and none of them asks the codebase to be globally healthy before you are allowed to merge a two-line fix.

What it costs

Three honest problems, and one of them is expensive.

The scoping is a heuristic and it will miss things. Playwright says so itself: “This is a heuristic and might miss tests, so it’s important that you always run the full test suite after the preliminary test run.” cargo-mutants has a sharper edge, documented plainly: “The diff is only matched against the code under test, not the test code.” A pull request that deletes a load of assertions and nothing else will sail straight through the mutation gate. Which, given what agents do to test files, is precisely the case you care about. Run the full suite nightly and never pretend the diff gate replaced it.

Coverage is not quality, at any scope. Inozemtseva and Holmes generated 31,000 test suites across five systems and concluded that coverage “should not be used as a quality target because it is not a good indicator of test suite effectiveness.” Patch coverage is a better-shaped number than project coverage. It is still not a measure of whether the change is correct, which is why the mutation gate is the one doing the real work here.

And it is a tax. DORA’s May 2026 modelling put the cost of AI adoption squarely on verification, with change failure rate rising from 5% to 6% in a modelled 500-person organisation, carrying $344,000 of modelled downtime, and concluded that the financial burden of adoption has shifted to governance. Those are modelled figures rather than measurements, but the direction is the finding: you are going to pay for verification either in CI minutes or in incidents, and CI minutes are much cheaper.

Try this week

Pick the file in your codebase you would least like to be wrong. Billing, permissions, whatever yours is.

Run a mutation test against it, once, by hand. Not in CI, not scoped, just once against that one file. The number that comes back is how much of that file your tests are actually holding, as opposed to visiting. For most people, on a file with healthy-looking coverage, it is a bad afternoon.

Then wire exactly one gate: mutation score, scoped to the diff, on that directory only, as a warning. Not the whole repository, not blocking, not all five gates. One directory, one signal, reported and ignored for a fortnight while you find out how noisy it is.

A gate you trust on one directory beats five you turned off in March.