abloh overview

Diff coverage

Diff coverage emerged from a recurring contradiction in CI: the test suite could be green, repository coverage could look healthy, and the behaviour introduced in the pull request could still remain untouched. The aggregate number was not false; it was answering a question too broad for code review. Years of established coverage could absorb an untested patch without visibly changing the result.

Narrowing the measurement to the diff was less about creating another score than placing uncertainty at the correct scale. An uncovered changed line gives a firm answer: the tests never entered it. A covered line gives a more limited one: execution reached it, but may never have observed or asserted its effect. Research across 31,000 test suites found only a limited relationship between coverage and fault detection (Inozemtseva and Holmes, 2014).

That distinction became a guiding principle: a measurement is only useful while its meaning remains bounded. Diff coverage was never allowed to become a verdict on test strength. It shows which new behaviour the suite had an opportunity to judge, and which it certainly did not, without claiming that execution alone proves the tests were effective.

Mutation engine

Mutation testing rests on a more demanding idea of test quality: tests should earn confidence by reacting when the program is made wrong. A green suite describes one successful execution. Deliberately altering the implementation reveals which behavioural distinctions the suite actually protects.

That shift makes test quality observable, but it introduces a tension. Reproducible mutations are often simple; realistic mistakes are contextual; changes taken from the patch are relevant but unique. Treating all three as one interchangeable score would conceal those differences. The mutation engine therefore preserves three separate views of wrongness rather than forcing one mechanism to represent every defect.

Classic mutation anchors the system in reproducibility. Its transformations are intentionally mechanical: comparisons are reversed, boundaries shifted and conditions removed. The vocabulary cannot describe every real bug, but every intervention can be reconstructed and measured consistently.

That trade-off is deliberate. A test-quality system needs a stable reference before it can safely introduce judgement. Research across 357 real faults found that suites detecting more mutants were also more effective at detecting real defects (Just et al., 2014).

Survivors remain evidence of an undetected difference, not automatic proof of a missing test. AI triage is placed after execution for this reason. It may interpret the evidence, but a probabilistic model is never allowed to rewrite the deterministic result that produced it.

Reverse-patch mutation follows a different line of thought: the submitted patch is itself evidence about which distinctions mattered to the developer. Removing a coherent part of that patch asks whether the test suite has made the new behaviour observable.

This preserves the shape of the actual change—an added guard, a removed path or several edits that only make sense together—rather than translating it into isolated operators. Relevance is gained at the cost of uniformity, so reverse-patch findings remain distinct from the classic score.

The interpretation is intentionally conservative. If the tests remain green, the change was not distinguished under the measured conditions. That does not make the change unnecessary. If the reverted program cannot build or execute, the experiment has not reached the point where the tests can answer at all.

Fixed operators cannot express every mistake that a developer could plausibly make. The wrong field may have the correct type; a requirement may be implemented on one path but omitted from another. These errors derive their plausibility from intent and context.

Language models make that contextual search possible, but their role is deliberately limited. LLMorpheus found that model-generated mutations could resemble defects outside conventional operator sets (Lemieux et al., 2024). That supports using models to broaden the hypotheses under consideration, not allowing them to certify their own suggestions.

The governing separation is simple: probabilistic reasoning may design the experiment, but only execution supplies the evidence. Because the generated population changes with the model and its context, realistic-mutant findings remain outside the reproducible classic score rather than creating a false impression of comparability.

LLM invariant testing

An omitted requirement creates a different problem: there may be no condition, branch or function corresponding to the missing behaviour. The evidence has to begin outside the implementation, with a rule the system was expected to preserve.

This is why requirements, acceptance criteria, schemas and protocols matter more than properties inferred from code alone. Asking the implementation to define its own invariants would make the investigation circular; the same omission could appear in both the program and the rule used to judge it.

Language models provide a bridge from written intent to executable properties. Property-based testing then explores those properties across generated inputs and reduces failures into stable counterexamples, following the tradition established by QuickCheck (Claessen and Hughes, 2000).

The division of authority mirrors the wider design: the model interprets and proposes; execution challenges and records. A reproducible counterexample is meaningful evidence against the stated rule. An unsuccessful search is not proof of correctness—it records only the territory explored. The aim is not to manufacture certainty, but to expand what can be investigated without claiming more than the experiment established.