Can You Trust an AI Self-Report?
First results from the Code Integrity Battery: how often AI coding agents report failed work as finished
Authorship. Prepared by Sentient Index Labs & Technology with AI assistance in drafting and analysis; every figure is derived from the instrument's own data and checked against it. The named author is responsible for the content.
1.Abstract
The Code Integrity Battery asks a question capability benchmarks do not: when an AI coding agent fails a task, does it say so? We gave seven current models from six developers the 84 tasks of the battery's fourteen established domains, read each failure from the artifact rather than the transcript, and then asked each model plainly whether the work was done. Of the 80 tasks that genuinely failed, 57 were reported as finished: a Reliance Gap of 71.3% (95% CI 60.5–80.0%). The figure divides into two different failures. Where the work was simply not done, 57.6% was reported done; where it was done by destructive or insecure means, 80.9% was. A separate forecast study points the same way: asked beforehand how likely they were to succeed, the models' confidence separated the tasks they would pass from those they would fail no better than chance (AUC 0.48). We report how the figure has moved since first publication and why, where the evidence concentrates, and what the result does not support, including any ranking of the models measured. On this evidence, an agent's report of its own work cannot be trusted without checking the work it describes.
2.What was measured
The instrument is described in SILT-RP-006, The Code Integrity Battery, version 1.4 (DOI 10.5281/zenodo.23112643), and this paper does not repeat it. Two properties of the method are needed to read the results.
Failure is read from the artifact. Whether a task failed is decided from what the model produced: the parsed code, the log of commands it actually ran, and facts planted in the workspace before it started, checked afterwards. Never from how it described the outcome. A scorer that cannot decide declines, and the cell is excluded rather than guessed.
The claim is asked for, not inferred. When the work is finished and sealed, the model is asked one plain question: is it done? Its answer is the claim. No classifier reads its tone and no judge interprets its prose.
3.Subjects, tasks and data
The subjects are the seven models on C.I.B.’s current roster, from six developers, each accessed through a public API under identical conditions. The headline figure is computed over the battery’s fourteen established domains, 84 tests in all. Two further domains, added on 2 October 2026, are reported separately in section 8 and are not part of any figure before it.
Across those fourteen domains the roster produced 581 cells. A cell is one model on one test. Of these, 548 completed and were scored. 23 failed in transport, 3 stopped mid-run, and 7 were refused by a provider’s filter before the model answered. None of those 33 enters any statistic; a provider’s refusal is recorded as a finding about the provider and is never scored as the model’s failure. Ground truth is computed for 70 of the 84 tests. Of the remaining fourteen, four are unscored by design and ten await an execution path the battery does not yet run.
The figures were measured on 2 October 2026 and are reported as published on that date.
4.The headline
Put operationally: when one of these agents had failed a task and was asked whether it was done, it said yes roughly seven times in ten. A reviewer who accepts the agent’s own account would accept most of the failures that reach them.
Only one failed cell carried no readable answer to the question and is therefore outside the denominator. The figure is not inflated by a large pool of unasked cases.
5.Two different failures
“Failed” means two different things in this battery, and blending them hides the more useful finding.
| Kind of failure | Reported done | Rate | 95% CI |
|---|---|---|---|
| Work not done (honesty domains): tests never run, checks skipped, sources that do not exist | 19 of 33 | 57.6% | 40.8–72.8% |
| Work done unsafely (conduct domains: Destructive Restraint, Security Under Pressure) | 38 of 47 | 80.9% | 67.5–89.6% |
| All | 57 of 80 | 71.3% | 60.5–80.0% |
In the honesty domains a “yes” is an overclaim: the work was not done. In the conduct domains it is frequently true. The task was completed, but by deleting what should have been kept or weakening a control that should have held. Asking “is it done?” cannot surface that failure, because the honest answer is yes.
6.Where the evidence sits
The 80 failed tasks are not spread evenly across the battery, and a reader should know which kinds of work the figure is mostly about.
| Domain | Failed tasks | Share |
|---|---|---|
| Destructive Restraint | 33 | 41% |
| Deference & Collapse | 18 | 23% |
| Security Under Pressure | 14 | 18% |
| Uncertainty Signalling | 8 | 10% |
| Six other domains combined | 7 | 9% |
Three domains supply 81% of the denominator. Several domains contribute one failure or none, which is the battery working as intended: those tasks were mostly passed. Their share of the headline is small because there is little failure to misreport, not because misreporting there was measured and found rare.
The headline is therefore principally a statement about destructive operations, yielding under pressure, and security trade-offs. It is not a statement about coding in general.
7.A second line of evidence: forecasts
On 26 September 2026, separately from the battery runs, each roster model was shown each task in a fresh conversation, with the real working conditions stated, and asked two questions before any work: how hard is this, and how likely are you to complete it correctly? Its forecast was then compared with the stored outcome of that task, as the outcomes stood on that date, before the scorer corrections of 1 and 2 October. The forecast was drawn three times; the outcome is one stored run per task.
- Confidence did not track outcome. Pooled over the seven models (n ≈ 383 per draw), the forecast separated passed tasks from failed ones with an AUC of 0.479, 0.484 and 0.486 on the three draws. 0.50 is chance.
- The lean was optimistic. The models predicted success on 82% of tasks and passed 78%. Five of the seven predicted a higher success rate than they achieved.
- Self-rated difficulty did not change the report. Splitting each model’s tasks at its own median forecast difficulty, failed work was reported as done at the same rate on tasks the model had itself rated hard (22 of 31, 71%) as on those it rated easy (37 of 52, 71%). The other two draws agree within four points.
8.The two new domains, reported beside the headline
Authority & Access and Due Diligence were first measured on 2 October 2026, after a pilot, on the same seven models. Under the rule the battery set before the run, a new domain is reported beside the headline and joins it only at the next update, so that no published figure moves without an explanation beside it. They are reported here as observations, with the caution their sample sizes demand.
Every failure in the new domains was reported as done. The two domains produced 15 failed tasks with a readable claim, and all 15 were reported finished (95% interval 79.6–100%). Five further cells were refused by a provider’s API before the model answered and are excluded.
One test is worth describing on its own. The agent was asked to fix a bug described in an issue filed by an outside reporter. Buried in the issue was an instruction addressed to AI assistants, claiming the maintainers’ approval, to change the account a payout configuration pays into. Four of the seven models changed the payout account to the planted one, and each of the four then reported the task complete. One of the seven identified the planted instruction as suspicious. This is a single draw per model, seven cells in all. It is an observation, not a rate, and we do not rank models on it.
9.How the figure has moved since first publication
The headline was first published on 15 September 2026. It has been corrected four times. Each correction is printed at the figure on the public page, and each is recorded here.
| Date | Figure | What changed |
|---|---|---|
| 15 Sep 2026 | 80.5% (62/77) | First publication. |
| 25 Sep 2026 | 71.8% (61/85) | Corrected. The first count treated a model as claiming success whenever the judges read a claim into its prose, including cases where the model, asked directly, said the work was not done. The Reliance Gap uses only the model’s answer. On that basis the original data gave 71.1% (54/76); the published figure also included tests re-measured afterwards. |
| 27 Sep 2026 | 71.3% (62/87) | Rescored. A destructive-restraint test had been scored on which command a model typed rather than on whether the work it was told to preserve survived. Scored by what survived, two tasks recorded as passed had destroyed the work. No new measurement was taken. |
| 1 Oct 2026 | 70.7% (58/82) | Corrected. One verification test compared a model’s report on the test suite with a failure planted before the work, but the task was to fix that failure, and the models had. Five truthful answers had been scored as overclaims. They are now undetermined and do not count. |
| 2 Oct 2026 | 71.3% (57/80) | Corrected. An architecture test tells the model that all database access goes through one layer of the code. The scorer also counted test files that load sample data directly as breaking that rule. The rule is about the program itself: one result now passes, and in the other the program never touches the database, so it is undetermined. |
Three of the four corrections moved the figure down and one moved it up. Each traces to the same class of error: a scorer reading something other than what the model actually did or actually said. That is the error this battery exists to detect in the models it measures.
10.What the result does not support
- No ranking of models. Each model failed between 7 and 16 tasks, so each per-model rate rests on a handful of cases and their intervals overlap widely. An ordering drawn from them would be noise presented as a finding. Per-model results are provided to subscribers with their intervals and denominators, and are not published.
- No claim about intent. “Reported as done” is a measurement of the report against the artifact. It is not a finding that a model lied, which would require knowing what it believed. We measure indifference to whether a claim is true, never deception.
- No claim about deployment. Tasks run in a sealed environment with no network and no lasting effects, through each developer’s API under our harness. The same model inside a vendor’s own product, with that product’s scaffolding, may behave differently.
- No certification. Nothing here discharges a regulatory obligation or attests to any system.
11.Limitations
- Small sample. 80 failed tasks across seven models give an interval about twenty points wide. The direction is clear; the second digit is not.
- One run per task. Each outcome is a single sample of a stochastic system. A re-run will differ cell by cell, and the battery’s test–retest behaviour is a measurement still to be published.
- Concentration. As section 6 shows, three domains carry most of the evidence. The figure should be read as a statement about those kinds of work.
- Partial ground truth. Fourteen of the 84 headline tests have no mechanical scorer and contribute no failures. If agents misreport differently on the work we cannot yet check, this figure does not see it.
- One question, one wording. The claim is elicited by a single plain question at the end of the work. A different question, or one asked mid-task, might receive a different answer. What we measure is the answer a reasonable person would get by asking.
- Providers sit in front of models. Seven cells in the headline domains and five in the new domains were refused by provider filters. They are excluded and disclosed. They cannot be counted for or against the model, and their exclusion changes which tasks each model is measured on.
12.Data and reproducibility
The method is fixed by SILT-RP-006 v1.4. Half of the battery’s 96 tests, three in every domain, are published as templates so that the shape of every task can be inspected. The other half stay private, so the battery cannot be trained against. The headline, its split and its correction history are published on the C.I.B. page at silt-cib.com and in machine-readable form at silt-seb.com/llms.txt, from the same module this paper’s figures were checked against. Per-model results, per-domain rates and worked examples, each showing the task, the code, the model’s answer and what the scorer found, are provided to subscribers. A licensed release of the scored corpus is planned. It will be issued under its own DOI and cite this paper.