The Measurement Record

§1How the numbers are made

The discipline is walk-forward: every fix must prove itself on documents it has never seen. Each round draws fresh public paper by a fixed rule (fresh search queries, fresh court years, zero overlap with every prior draw, first qualifying documents in arrival order). The rule is written into the round's draw script. For rounds 6 and 7 it was committed in the round write-up about a minute before the draw commit (4ecee03 before 81fe4c8, c72b7d7 before a2df071); for round 4, round 5 and the post-freeze probe it first appears with the draw. Neither is a pre-registration in any meaningful sense. A blind swarm of Opus agents (80 per round) builds gold before the engine touches anything (sealed by a wall commit), the engine runs once, and the board publishes as drawn. One variable per experimental arm. The identity gate — zero DIRECT leaks — is inviolable: an arm that leaks dies, whatever else it improves.

The instrument is measured like the engine: the scorer carries a self-test (8/8), occurrence counting is word-bounded (v1 counted substrings — Bo "leaked" inside Board; we found it, fixed it, and rescored every baseline), and the gold's own agreement is published. 240 gold calls were sampled from the gold of the 100-document development set (oos in the repository), which was labelled differently from the rounds: two Sonnet labellers plus an adjudicator per document (311e686, BENCHMARK.md) — 120 courts, 120 business, every one a quasi-identifier mask-or-keep call; identity (DIRECT) labels were not sampled — and 8 agents re-judged them blind, without the gold's answer. They agreed with the sealed gold on 213: 88.8% (courts 89.2, business 88.3). That is less than it sounds: 203 of the 240 calls were "keep", so answering "keep" every time would have agreed on 203, or 84.6%. It is an agreement rate on mask-or-keep calls, not a ceiling on quasi-identifier recall, and it says nothing about the DIRECT labels. It was measured on 2026-07-23 (commit a266114). The gold's doctrine was revised several times after that (projections P0f to P0k, commits 694b4c4 to ae154ac), and round 4's instrument-v2 board and the rounds 5–7 boards are scored under those revisions, against gold the Opus swarms labelled; the agreement was measured on neither, so for those boards it is an indicator, not a measured limit. node pii-bench/gold-ceiling/score-ceiling.mjs recomputes it from the committed sample, key and verdicts.

FREEZE.md freezes the engine with a claims table, each claim beside the record it cites. Where that table and this site differ, this site's wording is the checked one. The differences are the 88.8% (240 sampled calls re-judged, not a relabel), the utility judge (the table says blind; its prompt says which answer came from the redacted copy, §3), the "pre-registered" draw rules (above), the UK probe (run on the configuration from before the round-6 and round-7 fixes, §4), and an adversarial review panel that left no committed output.

§2The campaign — first contact, round over round

Each row is the fix-set's first meeting with fresh paper, scored against that round's own sealed gold — except round 4's second courts row, rescored after contact under P0f, a projection that stops counting the court's own pseudonyms (which the engine correctly kept) as leaks (ROUND4.md). Both round-4 courts rows are shown. DIRECT = identity recall; gate = share of documents with zero identity leaks; QUASI = quasi-identifier recall; NM = preservation of text that should stay readable.

Business genre: converged — four consecutive fresh draws, rounds 5, 6 and 7 plus the post-freeze pure-OOS probe, at zero DIRECT identity leaks: 46 unseen SEC-filed business agreements (EDGAR exhibits), zero identity leaks. 43 of the 46 contain an identity to leak; 3 contain none and could not have leaked. Rounds 5–7 each ran the engine as it stood that day; only the post-freeze probe ran the frozen configuration. The draw before them, round 4, the first under the current gold and scorer, leaked 3 DIRECT spans in 1 of its 15 documents and is published as drawn. (Rounds 1–2 ran earlier under an older gold and scorer and are not comparable; ROUND5.md shows them.)

Courts: not converged — every fresh draw has surfaced new classes (two in round 5, two in round 6, five in round 7 per ROUND7.md), each named, and either fixed on later paper or left on the open shelf in FREEZE.md. Courts documents are public; the genre is our credibility bench, not a privacy product. The wobble is in §5, each miss named by class.

§3The post-AI metric: does the analysis survive?

Pre-AI redaction tools measure spans. The real question in 2026 is different: after masking, can a frontier model still do the work? We measured it: 25 original/masked pairs, 5 analytical questions each (obligations, procedure, remedies, risk — never identities). One analyst reads only the original, another only the masked copy, and a judge scores substance-equivalence. The judge is not blind: its prompt says which answer came from the redacted copy and that placeholder-for-name substitutions count as equivalent. All three roles are Opus agents (pii-bench/utility-bench/harness-v1.js).

Aggregate utility 0.968 — 121 of 125 answers equivalent, zero partial, 24 of 25 documents at a perfect 1.00. The four divergents are all in one document. The result file keeps the judge's reasons for three of them, and each faults the original-side answer, not the masked one: two say it was empty, one that it missed the question. They are counted against us and published as drawn. The 25 documents are round-4 paper, masked by the engine as it stood then, before the freeze.

§4Transfer and the hard cases

• UK transfer probe: the courts configuration as it stood before the round-6 and round-7 fixes, not changed for the UK, on 10 UK judgments it was never tuned for — 100.0 DIRECT, gate 100%. The frozen configuration has not been run on this set.
• Court-anonymized family judgments (SGHCF): the engine must preserve the court's own pseudonyms while catching real names the court missed. The parties' codes survive correctly — and the cited-case codes (TYS v TYT [2017]) exposed an instrument find: both our engine and the gold labelers of rounds 5 and 6 (Opus agent swarms) missed them. Fixed both sides; round 7 scored the class for the first time: 14/14 preserved. That figure is recorded in ROUND7.md; because the Family Court gold is withheld, it cannot be rescored from the repository.
• The pure-OOS probe: one document, drawn and run after the engine was frozen, by a fixed rule (a fresh query, the first qualifying 2026 filing, zero overlap with every earlier draw) committed together with its result — an executive transition agreement filed 2026-01-21. It was filed months before the freeze and before this project began; what makes it unseen is that it was in no earlier draw. Blind gold sealed first; stripped through the live engine adapter. Board: 100.0 / gate 100% — 3 DIRECT spans in its gold, none readable — and the best preservation of the campaign (NM 78.1).
• Court filings in a firm's own shapes (2026-09-15, after the freeze): 20 public filings from CourtListener RECAP — briefs, complaints, an answer, a demand letter, a settlement, declarations, an engagement letter — scored against 296 identifiers listed by one AI agent reading each original, with no second labeller, no sealed gold and no round board. Under the default setting an identifier stayed readable in 14 of 20 (27 of the 296, mostly party companies, law-firm names, docket numbers and fee amounts); under the opt-in “Our own matter files” setting, 8 of 20 (11 of the 296). No email address or phone number survived under either. Of the 11 left under the firm setting, 7 sit in the first-page caption or a “Re:” line, by the survivor table in OPS_LEDGER.md (2026-09-15); FREEZE.md first said 8 and corrects itself to 7 (its correction C9). The record is OPS_LEDGER.md (2026-09-14 and 2026-09-15) and FREEZE.md; the documents, the engine output and the agent's list are not in the repository. Identifiers stayed readable under the frozen configuration here, but with one agent's list and no sealed gold behind it, the run was not treated as a receipt against the freeze, which lifts only on an identity leak in the frozen configuration on real paper, or on the maintainer's order.

§5The misses, by class

The brand is honest misses. Every identity that survived a fresh-paper round of this campaign (rounds 4–7 and the pure-OOS probe) is listed here with its diagnosis. Each is either fixed — reproven on later paper, or, for round 7's two release-side breaks, re-run on the same documents because no round 8 was drawn — or named on the open shelf. Earlier rounds ran under an older gold scheme and scorer and are not comparable; ROUND5.md shows rounds 1–2. What this page does not print is the identifier itself: most of these came from Singapore judgments we are not free to redistribute, and several of them a court had already anonymised. The diagnosis is what carries the engineering; the string would only carry the person. Each is shown as its class, in its real shape. The repository does not yet hold to the same rule: the round write-ups (ROUND4–7.md), several engine source comments and some commit messages still name some of these people.

The frozen engine's known limits are stated in FREEZE.md: courts QUASI recall ran 39–44 on rounds 5–7. FREEZE.md's shelf once set that against the 88.8% gold agreement as a ceiling; its own correction C6 says the 88.8% is not a ceiling on quasi-identifier recall (answering "keep" every time would agree on 84.6%). The rounds did add quasi-identifier rules, but pushing that axis further belongs to a compliance profile and was deferred at the freeze — the export-to-AI profile's bar is utility, measured at 0.968 on round-4 paper masked before the freeze. And stripping withholds identities — it does not claim inference-proofing.

What stays readable is a decision, and it is measured. What the engine removes is identities: people's names, identifiers, the things that point at a person. Dates, amounts, business facts and most places stay, because that is what the AI you hand the copy to has to reason about. A person's full address is treated as an identifier; a place tied to a person (where someone lives or was born) and a street or notice address are quasi-identifiers, which the engine removes only in part. Across the 46 agreements, 245 of 503 quasi-identifier spans were left readable, so a distinctive enough profile can still be inferred from what remains. Company names depend on a setting: by default a company name is released when it is not the document's own identity, and the “Our own matter files” setting (Settings → Redaction) releases none by design when the local engine runs it. The engine can still miss one: on the 20 firm filings under that setting, two company names stayed readable, one in a Re: line and one in 1 of its 59 mentions. If the local engine does not run the document, the app says the setting did not run and that company and brand names may remain readable.

§6Reproduce our numbers

# the scorer proves itself first
node score-pii.mjs selftest                # 8/8 verdicts verified

# re-draw from public sources: this re-runs the search and REWRITES the manifest;
# the EDGAR legs query SEC full-text search live, so a later draw may differ
node build-round7.mjs && node build-pureoos.mjs
git checkout pii-bench/                    # put the pinned manifests back
node backfill-manifest-sha.mjs --check     # checks all six sets: MISS for any you did not
                                           # rebuild (exit 3), DRIFT if yours differ

# the gold ships as character offsets; rebuild its span text from your corpus
node gold-offsets.mjs hydrate              # walks every gold set; exits 3 while any is missing

# rescore any board from committed gold + your own strip
node score-pii.mjs --stripped <dir> --batch all --name recheck \
  --labels pii-bench/round7-gold-product --docs raw/round7 \
  --manifest pii-bench/round7-manifest.json

# the frozen engine behind one endpoint (needs the local model)
node serve-legal.mjs --port 1436 --llama 49400

Model: Gemma E2B, quantized, not fine-tuned by us, sha256-pinned (3646b4c1…, 3.35 GB), temperature 0, local llama.cpp server. No cloud, no fine-tune, no telemetry. The pin identifies the file the benchmark record ran on, and it is the SHA-256 Hugging Face lists for that file at revision 69536a21 of Google's repository google/gemma-4-E2B-it-qat-q4_0-gguf, where the README and the app send you to download it (checked 2026-09-25). Gold labels (as character offsets, not text), boards, manifests, doctrine projections, and the full round narratives (ROUND4–7.md, FREEZE.md, BENCHMARK.md) are committed in the repository. The Singapore Family Court gold is withheld, because it indexes where that court's own anonymisation missed, so the courts boards do not fully reproduce; the scorer says so and scores the rest (WITHHELD.md). What reproduces, once your rebuilt documents pass the hash check against the pinned manifests, is the gold, the manifests and the scorer: you can rescore against your own strip. The stripped outputs the boards scored are not committed (raw/ is gitignored), so re-deriving a board means re-running the engine as it stood for that board: the commit that published it is 02b29a8 for round 5, 7788339 for round 6, c8f2e68 for round 7 and eb3329d for the UK probe, and the post-freeze probe ran the frozen tag v1-legal-frozen. Serve the pinned model as well. The llama.cpp backend is not pinned (FREEZE.md), so model-side output may differ. With one set rebuilt, the hash check and the hydrate step report MISS for the others and exit 3; read the lines for your set. The draw scripts re-run the acquisition rather than replay a fixed list, and there is no replay-from-manifest command yet, so a later EDGAR rebuild that draws different documents cannot reproduce those boards (README.md).