abloh01
Sonar developer survey · n=979About
Overview
abloh is the verification layer for AI-written code. When AI writes both the code and its tests, both can be built on the same mistaken assumption. Passing tests show only that the suite is green, while coverage shows which parts of the code your tests exercised. Neither shows the quality of your tests - whether they would detect incorrect behaviour.
On every pull request, abloh runs inside your CI, combining mutation testing with LLM invariant testing. Invariant testing works from your requirements rather than the implementation, checking the rules your code was expected to preserve. Mutation testing measures which changed lines your tests execute, then tests the tests themselves. Classic mutation plants mechanical faults and counts how many your suite detects, while LLM realistic mutants plant plausible mistakes a developer could make and a user would encounter. Their findings are then triaged by AI to separate real gaps from noise and propose the missing tests.
Why now.
AI has made code and tests faster to produce. The work of verifying them has not kept pace.
42% today → 65% by 2027of committed code is generated or significantly assisted by AI, according to developers
02
Dipongkor et al., 2026Agents often leave the tests unchanged
03
Sonar developer survey · n=1,149Developers still do not trust the result
When AI writes both a change and the tests meant to verify it, both can repeat the same mistaken assumption. abloh gives every pull request an independent check before merge.
Why abloh exists.
Diff coverage
Diff coverage emerged from a recurring contradiction in CI: the test suite could be green, repository-wide coverage could look healthy, and the behaviour introduced by a pull request could still be untested. The aggregate number was not false; it was answering a question too broad for code review. A mature, well-covered codebase could absorb an untested patch without visibly changing that percentage.
Narrowing the measurement to the diff was less about creating another score than placing uncertainty at the scale of the change. An uncovered changed line gives a firm answer: the test run never executed it. A covered line gives a narrower one: the test run executed it, but the suite may never have checked whether its effect was correct. Research across 31,000 generated test suites from five Java systems found only a low-to-moderate correlation between coverage and mutation-based fault detection once suite size was controlled for (Inozemtseva and Holmes, 2014).
That distinction became a guiding principle: a measurement is useful only if its meaning remains bounded. Diff coverage was never allowed to become a verdict on test strength. It shows where the suite had an opportunity to check the effects of a change, and where it certainly did not, without claiming that execution alone proves the tests were effective.
Mutation testing
Mutation testing rests on a more demanding idea of test quality: tests should earn confidence by reacting when the program is made wrong. A green suite describes one successful execution. Deliberately altering the implementation reveals which behavioural distinctions the suite actually protects.
That shift makes test quality observable, but no single mutation can represent every way software becomes wrong. Mechanical mutations are reproducible but often simple, while realistic mutations reflect context but require judgement.
Classic mutation
Classic mutation turns test quality into a controlled experiment. It applies a fixed vocabulary of small, mechanical changes, such as reversing comparisons, shifting boundaries and removing conditions. If the suite fails, the change was detected; if it remains green, the mutant survives. Across 357 real faults in five open-source Java projects, mutant detection was positively correlated with real-fault detection, independently of code coverage (Just et al., 2014).
Controlled does not mean conclusive. Mechanical generation also produces noise: some mutants are behaviourally equivalent to the original program, several may express the same underlying weakness, and others may alter behaviour irrelevant to the intended contract. A survivor therefore establishes only that the suite did not distinguish the altered implementation from the original. Whether that difference reflects a meaningful test gap is a separate question.
This is why mutation testing requires triage. Execution identifies the survivors; an AI model then evaluates them in context, separating meaningful test gaps from noise.
LLM realistic mutants
Some mistakes remain syntactically ordinary. A value may come from the wrong field while retaining the correct type; a requirement may be enforced on one path and omitted from another. What makes these changes faulty, and useful as mutants, is their relationship to the surrounding logic and intended behaviour.
Language models make it possible to construct mutations within that context. In a case study of 40 real bugs, LLMorpheus generated mutants resembling real defects, including faults that traditional operators could not reproduce exactly (Tip et al., 2025).
Realistic mutation turns that broader search into an adversarial test. The model constructs a plausible, context-specific fault, and the altered program is then run against the existing suite to see whether the mistake passes unnoticed.
LLM invariant testing
An omitted requirement creates a different problem: there may be no condition, branch or function corresponding to the missing behaviour. Detecting it requires an invariant drawn from outside the implementation: a rule expected to remain true across a range of inputs or program states.
Requirements, acceptance criteria, schemas and protocols provide that independent account of intended behaviour. Inferring properties from the code alone would make the investigation circular; the same omission could appear in both the program and the rule used to judge it.
Language models bridge written intent and executable properties. Property-based testing then generates varied inputs, checks whether each property holds and, when one fails, shrinks the input to a simpler counterexample, following the QuickCheck tradition (Claessen and Hughes, 2000; Hughes, 2007). This makes missing behaviour testable even when the implementation contains no trace of it.
What abloh cannot catch.
abloh can change existing code and prove whether your tests notice. It cannot inject a fault into behaviour that was never implemented.
To find missing cases, abloh needs a clue from the ticket, types, invariants or surrounding code. If the behaviour is described nowhere, abloh cannot know it was intended.
abloh also cannot reproduce failures caused solely by production infrastructure, unavailable external services or unsupported concurrency schedules.