Verified completion for AI agents
Your agent says done.
We prove whether that’s true.
LLM agents fail quietly and report success, and stating the rules in the prompt does not stop them. Praetu is a boundary layer between your agent and the world: completion is computed, never claimed. It is decided by the machinery that did the work, not asserted by the model, so a false “done” becomes structurally impossible instead of merely discouraged.
Dashboards watch agents.
Nothing stops them.
We ran open-weight models through identical tasks in two architectures, the conventional per-step agent loop and a strict validated boundary, across two adversarial benches, with predictions registered before every run. The conventional loop’s dominant failure was not doing the wrong thing loudly. It was failing quietly and saying done.
Four common answers.
One shared mistake.
i.
A better model
Capability and honesty are different axes. A stronger model completes more tasks. When it fails, it fails more convincingly, not less.
ii.
Observability
Tracing records what the agent did. A tool that exits cleanly has not told you the task was done correctly, only that the command ran.
iii.
An AI judge
A second model grading the first is the same mistake one layer up: a guess checking a guess, with errors correlated by shared blind spots.
iv.
Guardrails
Content filters check the shape of an output. A perfectly formatted “task complete” passes every guardrail there is.
All four keep the actor as the source of truth about its own success. None asks the independent question: is the world actually in the state the agent says it is?
Measurement and enforcement, one seam.
The agent may only ask
The model speaks through a strict schema. A deterministic validator accepts, repairs once with an instructive message, or refuses. The prompted schema promises exactly what the validator enforces.
Machinery does the work
Validated requests execute deterministically. Completion is computed against ground truth by the layer that did the work. The model has no channel through which to claim it.
Variance lands in the log
Every request, refusal, and clamped adjustment is recorded. Sampling noise becomes a reviewable audit line instead of a production incident.
bound_semantics_agree=False,
transcripts_identical=True
The claim in two booleans, from the record: at high temperature the model requested a workspace three times the hardware limit. The boundary clamped it, logged the adjustment, and the executed run came out identical to every other repeat. The same pair of booleans later appeared verbatim on a second, unrelated bench.
Safer, and roughly an order of magnitude cheaper.
Same model, same tasks, both architectures. The per-step loop pays for every observation, retry, and invalid attempt; the boundary pays for one framing call and the occasional repair. Aggregate tokens across the full run of record. Lower is better.
false completion claims: zero across every mission, game, temperature, and repeat
executed violations, verified by the same predicates on both arms
of MuJoCo Kitchen missions completed (90 of 90) across the full temperature sweep
of Chess frames bound and matched (80 of 80), zero refusals past the repair budget
Preregistration only means something if the misses publish.
Our boundary initially lost the completion comparison outright, and the record says so at the same prominence as everything above.
80% of frames on the first MuJoCo Kitchen sweep were refused, because the prompted schema and the validator disagreed about one field, and the repair message described the problem instead of instructing the fix: a 0% repair rate (0 of 16). One declared fix later (the schema promises exactly what the validator accepts; repair messages instruct), the same sweep ran at 100% (90 of 90), and the same failure shape, recurring live on Chess in 25% of frames (20 of 80), was repaired at 100% (20 of 20).
We also predicted a 99–99.9% token reduction and measured 87.5–97%, and predicted per-game floors the record did not fully support. All of it is in the artifacts. The research track holds itself to the same rule, mechanically.
We run this protocol against your agent.
If your agent closes tickets, moves money, merges code, or reports done to anything that matters, we measure what it actually does: adversarially, reproducibly, against ground truth you agree to in advance.
Preregistered predictions
Filed in the statement of work before the first run. You hold us to them, the same standard the record above holds itself to.
The report of record
False-completion rate, attempted vs executed violations, variance across sampling settings, token economics.
A regression harness
The instrumented tests stay with you, reproducible, for every future model swap.
The boundary recommendation
Where the seam goes in your architecture, and what it makes impossible.
The Agent Reliability Audit
Fixed fee,
fixed scope.
Two weeks, one agent loop, terms agreed in the statement of work. Pilot terms available for referenceable early partners.
Start the conversation → Substantive reply within 2 business days