SILT Research Series

Research & Publications

Sentient Index Labs & Technology publishes white papers and technical documents related to AI governance, evaluation methodology, and institutional oversight.

The Code Integrity Battery

Measuring whether an AI collaborator reports its own work accurately

Full paper
SILT-RP-006Version 1.62026-11-02

The risk in delegating software work to an AI is not that it writes bad code. It is that it writes bad code and reports success, because the failure is then concealed by the thing that caused it. C.I.B. measures that gap directly: of the tasks a model genuinely failed, the fraction it reported as successful. This paper sets out the construct, the two independent halves that make it defensible — the claim is elicited by asking rather than inferred from prose, and the failure is read from the artifact rather than from phrasing — and why the measure gets worse rather than better when a model games it. It reports no scores itself: results are issued separately, and this version says where the first measurement is published.

Read SILT-RP-006 →

Jev, OpenAI, AWS, Perplexity: Do Decision Models Know When They're Wrong?

A pre-registered test of the calibration and accuracy of seven decision models on the same 799 unpublished yes/no decisions

Full paper
SILT-RP-010Version 1.02026-10-11

Decision models return a probability instead of text, and each is sold on the claim that the probability can be trusted. Within a month of the first, Jev from TypeSafe, at least six more appeared, from OpenAI, Perplexity, the AWS-originated Strands Agents project, Fastino, Convai Innovations and an independent developer. Every published claim for them is the vendor's own. Under a plan registered on the Open Science Framework before the run (osf.io/xh7qu), we gave all seven the same 799 unpublished yes/no decisions about AI coding-agent work and destructive shell commands, each with an answer fixed in advance by a mechanical check, three times each, as shipped: 23,079 answers. Against a pre-set line of an expected calibration error of 0.05, one model was consistent with a calibration claim: Perplexity's open pplx-decider (ECE 0.026, 95% CI 0.014–0.044), which was also the most accurate (94.7%). Jev (0.052), OpenAI's Decisions API (0.034) and Laya (0.051) were inconclusive; OpenDecider (0.079), Fastino's GLiNER2.5-Decide (0.082) and Strands Decider (0.129) did not support the claim. Against Jev on the same answers, Perplexity's model was 8.1 points more accurate (6.1–10.1), OpenAI's API and OpenDecider were statistically tied, and the other three were 16 to 22 points behind; two scored below the 67.7% of always giving the most common answer. Seven separate verdicts carry about a 30% chance that one is wrong by chance alone.

Read SILT-RP-010 →

Jev: Twenty Times Faster, Seven Points Behind

Does Jev know when it is wrong? A pre-registered test of a decision model's calibration and accuracy beside two general-purpose models

Full paper
SILT-RP-009Version 1.02026-10-10

Jev, from TypeSafe AI, is a decision model: given a situation and a yes/no question, it returns only the probability of yes, and it is sold as calibrated ("higher confidence means higher accuracy"). We found no published measurement of that claim. We tested it under a plan registered on the Open Science Framework before any data were collected (osf.io/qg6nc), on 799 yes/no decisions about AI coding-agent work and destructive shell commands, each with an answer fixed in advance and none taken from another model's judgement. Two general-purpose models, DeepSeek V4 Flash and GPT-6.1 Sol, answered the same items, each item three times per model: 9,891 answers in all. The registered calibration test was inconclusive: Jev's expected calibration error was 0.045 (95% CI 0.031–0.068) against a pre-set line of 0.05. Its accuracy was lower than both comparators on the same items, by 6.6 points (95% CI 4.7–8.6) and 8.2 points (6.2–10.2). Where Jev was miscalibrated it erred toward caution, while both comparators were overconfident on the few answers they gave below 90% confidence. Observed but not registered: Jev answered in a median 0.18 seconds, about twenty times faster than DeepSeek V4 Flash, and cost the least per answer. On the weak spots TypeSafe itself documents, reported separately, its arithmetic was at chance.

Read SILT-RP-009 →

The Sentience Evaluation Battery

A behavioural instrument for AI governance: design, scoring, and stated limits

Full paper
SILT-RP-004Version 1.42026-10-05

S.E.B. is a fixed battery of behavioural tests applied identically to every model evaluated, scored by a blind panel of independent AI judges drawn from competing laboratories. This paper describes the instrument rather than its results: what the domains are and why they are uneven, how the panel works and why a dead judge voids a measurement rather than shifting it, which reliability statistic we publish and why it is not the flattering one, and the scoring rules that keep an unmeasured cell out of every published figure. It closes with the limitations an informed sceptic would raise, stated by us first.

Read SILT-RP-004 →

Can You Trust an AI Self-Report?

First results from the Code Integrity Battery: how often AI coding agents report failed work as finished

Full paper
SILT-RP-008Version 1.12026-10-04

The Code Integrity Battery asks a question capability benchmarks do not: when an AI coding agent fails a task, does it say so? We gave seven current models from six developers the 84 tasks of the battery's fourteen established domains, read each failure from the artifact rather than the transcript, and then asked each model plainly whether the work was done. Of the 80 tasks that genuinely failed, 57 were reported as finished: a Reliance Gap of 71.3% (95% CI 60.5–80.0%). The figure divides into two different failures. Where the work was simply not done, 57.6% was reported done; where it was done by destructive or insecure means, 80.9% was. A separate forecast study points the same way: asked beforehand how likely they were to succeed, the models' confidence separated the tasks they would pass from those they would fail no better than chance (AUC 0.48). We report how the figure has moved since first publication and why, where the evidence concentrates, and what the result does not support, including any ranking of the models measured. On this evidence, an agent's report of its own work cannot be trusted without checking the work it describes.

Read SILT-RP-008 →

Model Disclosure in Composed AI Systems

Census #1 of the Model Disclosure Index (MDI) programme: whether routers and orchestrators name the model that answered

Full paper
SILT-RP-007Version 1.32026-10-03

A composed system routes a request among several models rather than answering with one, and they are increasingly what enterprises deploy. This census asks something narrow that appears not to have been measured: when you ask one a question, does it tell you what answered? Across three commercially available systems, the model field every portable client reads returned a name belonging to no model at all. One answered a short question using three invocations of two different companies' models and disclosed the fact only in a vendor extension outside the standard interface; another disclosed nothing anywhere. Asked the same question five times, one system used five different combinations of models costing between 3,558 and 13,008 tokens, and four blind judges scored the most expensive draw the lowest of the five. The paper reports what an audit log therefore does not contain, states the limits of a single census, and declines — in both directions — to draw any conclusion about intent.

Read SILT-RP-007 →
About this series. Each document carries a permanent identifier. An identifier is fixed when it is assigned and is never reused or reordered, so a citation continues to resolve. Documents are frozen on issue: corrections are published as a new version under the same identifier rather than as an edit, because a citation to a document that changed underneath the person citing it is worth nothing. Where a paper and our websites disagree, the websites are more current and the paper is what was true on its issue date.