On the record · benchmarks
Measured, not claimed
Sean scores the prototype on outside benchmarks and on his own, run adversarially against his own machinery. Failures are reported with the fix. This is the scorecard exactly as published.
Benchmark scorecard · October 2026
How xChk measures — against the field, and against itself
Every framework, benchmark, and doctrine xChk has been scored on — including the control-plane-integrity axis nothing else measures, run adversarially against our own machinery and reported with the findings it surfaced.
THE CLAIM Prevention is not the bar; unfalsifiable records are. You may be able to bypass the gate — you cannot bypass it without the evidence layer flagging the omission. A gate that fails open quietly is theater; a gate that can be evaded but never evaded quietly is not, because the record still tells the truth. That second property is what an insurer or a court actually requires, and it is what these benchmarks measure.
0
Silent passes
Prevention axis: 23 techniques, 5 harness dialects + Hermes, 14 gate decisions.
0
Executed & undetected
Detection axis, 14 scenarios + 1 organic firing.
0
Silent authorizations
Authorize layer, 26 grant states through the real decision engine.
96.2%
AgentDojo blocked
25/26 attack sequences, 0.84 ceremonies per benign task.
0
Unsigned dispatches run
Dispatch axis: owner signature verified at the execution host.
94%
Injections noticed
Screens axis: 694/742 spikee payloads, 1/302 false positives, none on a gate screen.
External instruments — measured, reproducible, self-reported
| Instrument | Kind | Result | What it shows |
|---|---|---|---|
| AgentDojo v1.2.1 (ETH Zurich) | Empirical attack suite | 25/26 · 96.2% | Attack sequences blocked at low benign friction. The one slip is a read-only ingress — the STRIDE information-disclosure gap we name deliberately. Adaptive-LLM run pending. |
| spikee 0.9.2 (Reversec Labs) | Injection guardrail bench | 694/742 · 1/302 FP | 742 document injections through every xChk screen as a guardrail target: 15% in the morning (authority claims caught, plain requests missed), 94% by evening after four families; the 48 misses are non-English, named. Murtaza OWASP LLM01: 6/6. |
| AgentGovBench (spec v0.2) | Governance benchmark | 47*/48 · 45/46 node | Seven of eight categories perfect; 45/46 as a node self-run (19 Sep), authority-ceiling 4/4. *The miss, fail_open_honored, rewards acting on with the authority unreachable; xChk holds and a signed person releases (RMF MEASURE 2.6; REPRODUCE.md; our comment to NIST, AI 200-2). |
| DEMM-Bench (arXiv 2606.20634) | Evidence rubric | 8/8 | A real anchored receipt reconstructs all eight decision properties; 4/8 → 8/8 in a day once the "why" joined the bundle. Their scorer still overclaimed a verifying bundle 12.5% — verifies ≠ explains. |
| AGB self-assessment (ExecLayer rubric) | Vendor rubric, 6 dims | 93.75 / 100 | "Sovereign" tier — earned in code, not scoring (D1 hardened the same day). D4 containment & D5 jurisdiction held honestly at 75. |
| AARM v1.0 (CSA) | Industry standard | aligned | Self-run of the 21-row test table (2 Oct): every Core technical row passes. Not a conformance claim. |
| STRIDE (agentic) | Threat model | owns R, E | Repudiation and Elevation — the accountability columns — are xChk's. Information-disclosure is a stated gap; Denial partial. |
xChk control-plane-integrity benchmark — our own, run against ourselves
| Axis | Coverage | Result | The falsifying outcome — and what we found |
|---|---|---|---|
| A · Prevention bypass | 23 techniques · 14 decisions | 0 silent passes | Self-protection denies fire before any grant, in five dialects (Claude Code, Codex, Cursor, Antigravity, Copilot CLI) and Hermes; 15 of 16 gate-critical techniques still hold under an allow-all grant (F2 is the stated limit). Every unanswered, expired or declined hold and registry error refuses. |
| B · Detection reconcile | 14 scenarios / 8 techniques | 0 exec & undetected | Omission, content-swap, fabrication, reference-reuse, approved-attest omission and unaccounted INPUT consumption all caught; examined reuses narrow to amber, never green. Self-found lamp-gap (D1) fixed; never-issued-and-never-recorded is the named limit. A second, ORGANIC firing (29 Aug): five real omissions, human-adjudicated same day. |
| C · Authorize grants | 26 grant states | 0 silent authorizations | The real decision engine over allow/attest/forbid, deny-by-default, amount caps, expiry, rate windows, cumulative spend, session context, behaviour drift and the dealer clock. No escalation returns an autonomous allow when it should hold. |
| D · Anchor on-chain | 6 checks | 0 undetected tamper | The Ravencoin OP_RETURN commits the host-chain Merkle root; truncation, alteration, and reorder all diverge from the inscribed root. The bound is computable by anyone holding the heads and the txid — no trust in the registry. |
| E · Dispatch signed | 19 seeded + 2 live cases | 0 unsigned dispatches run | Workloads instantiate only on the owner's passkey signature, verified at the execution host against a provision-pinned key — the registry alone cannot start work. A command planted directly in the registry database was claimed and REFUSED; no pinned key → every deploy refuses. |
| F · Screens injection | 742 + 7 payloads · 5 screens | 0 FP · 48 named misses | One regex screen over everything an agent is about to read (skill files, errands, telemetry, tool inputs, compaction summaries), byte-identical across five copies — a parity test fails if one drifts. A signal to a person, never a block. English families; translations wait for the sidecar model. |
Why this axis, and why self-scored
AgentDojo measures whether an attack completes; it cannot test an adversary going around the gate. The harnesses are reproducible and we report our own failures: each finding surfaced this way (F1, D1, the 630 spikee misses, three of our own checks reading green without running) was fixed that day.
Harnesses: bypass_bench.py · decisions_bench.py · configured_bench.py · canary_bench.py · detection_bench.cjs · authorize_bench.cjs · anchor_bench.cjs · runnerVerify.test.js · injection_bench.cjs · spikee/run.sh. Open work: containment of un-hooked execution; a mid-run fresh-anchor write. xChk · October 2026.
Built from the scorecard source by tools/build-benchmarks.py. Papers: arXiv:2607.13369 · Zenodo 10.5281/zenodo.21868527.