We publish how the numbers were made.
This page carries both markets, measured under one protocol — Korean public procurement (나라장터) and U.S. federal (SAM.gov). What was measured, how, what the pipeline missed — enumerated one by one — and where the measurement itself falls short.
- 87.4%
- Korea (나라장터) — requirement-extraction recall, frozen exhaustive golden set of 2,013 items (1,985 valid after removing 28 invalid labels)
- 93.2%
- U.S. federal (SAM.gov) — requirement-extraction recall, exhaustive golden set of 1,481 items
- 96.5%
- Korea — machine-certain judgment precision, after adjudicating every flagged case
- 103 / 103
- U.S. federal — machine-certain flags all correct in the latest benchmark run (three-vote unanimous judgment)
- 99.4%
- HWP·PDF table preservation — 500 attachments from one day of 나라장터 notices (2026-09-08), preferred format, parse errors excluded
Both markets follow the same procedure. The answer key is authored exhaustively and independently, with no access to the pipeline output, and every item carries a verbatim anchor that is machine-verified against the source. The key is frozen in a commit before the extractor runs, and after scoring, every item the pipeline missed is adjudicated one by one — nothing is left as a mystery.
When we change the extractor, we re-score against that same frozen key. So the before-and-after figures on this page are scores on the same exam, not a new exam with a friendlier denominator.
Korean public procurement (나라장터)
Two real, published negotiated-contract service tenders (8 documents, table-preserving parse). An adjudicator authored the full requirement set — 85 and 88 items, 41 of them critical — while barred from seeing any pipeline output, so the answer key could not be shaped by what the pipeline happens to extract.
Every item carries a verbatim anchor into the source text, and all 173 anchors pass machine verification. Table rows are included as items — tables are where Korean tender documents put the substance.
A single extraction pass scores 82.1% recall against the golden set. Adding one critic pass — a second call that only hunts for requirements the first pass missed — lifts it to 92.5%, beating an equal-cost union of two independent passes (88.4%). That is the configuration we ship: one extra LLM call per document.
The critic over-extracts by design; the deterministic grounding layer absorbs it — every added item, grounded rate 100%.
Real bid submissions are exempt from disclosure in Korea, so they cannot be obtained. Instead: for three real published tenders (engineering design, solar installation works, security services — HWP and PDF), we generated fully compliant bid packages, then injected 45 defects with seeded, fixed mutations — so every defect label is certain — plus 15 harmless-rewording traps.
The 18 / 19 figure on the verification page comes from a separate test — the adversarial corpus: packages seeded with 19 subtle defect types plus 2 clean packs for false-positive measurement, re-run whenever the prompt or model changes. The latest Korean run (2026-08-06) caught 18 of the 19, 17 of them machine-certain. It uses a different corpus and defect set from the 45 injected defects below, so the two numbers are not comparable with each other.
Machine-certain precision: 96.5% after adjudicating every flagged case. The deterministic layer caught every missing document, every identifier tamper, and every total-amount tamper — without LLM involvement.
Every miss was adjudicated individually — none are mysteries. Of 45 injected defects, the judgment layer missed 9:
- 3 — the notice figure lived in a table the (then-current) parser dropped; the table-preserving parse path since became the default
- 2 — the extracted requirement summary omitted the number needed to catch the deviation (granularity, not judgment)
- 2 — genuinely lenient judgment calls: a deleted section still voted “covered”. These are the real judgment failures: 2 of 45.
- 1 — an upstream extraction miss propagated downstream
- 1 — label turned out invalid: the deleted content was covered by another document
Also disclosed: 2 of the 15 harmless-rewording traps drew a “partial” flag (false positives), and the extraction residue clusters in edge cases such as authoring instructions inside a proposal’s table-of-contents table.
All 15 packages were verified twice, independently. The final pass/block posture agreed 15/15. What flickers between runs is confined to borderline grades — “is this commitment strong enough” — never the defect-anchored findings (missing documents, identifier and total mismatches, dropped requirements).
For partners and auditors: the comparison is against generic document tooling and profile-vs-criteria checkers, not against any named vendor.
| Table extraction from HWP/PDF | drops 85–87% of table cells (measured on 214 notices) | preserves 99.4% of tables (500 attachments, one day) |
|---|---|---|
| What is judged | company profile vs. scoring criteria | submitted documents vs. each requirement in the notice |
| Confidence | a single model answer | the same question asked three times, all answers agreeing + verbatim grounding; anything less is flagged for review |
| Before delivery | — | seven automated gates before delivery |
| Not measured here | — | evaluation scoring, performance interpretation, real vendor proposals (packages are synthetic) |
Numbers in this table use the same standards as the sections above; each has its own measurement page or note. Reproduce the tokenizer measurement (public repository) →
- The bid packages are synthetic. Real vendors’ proposals differ in style, structure, and error distribution — figures here are benchmark performance, not a guarantee on your documents.
- Judgment-of-merit factors — evaluation scoring, how track-record rules are interpreted — are out of scope; only document, figure, and commitment-level checks were measured.
- With 45 defect labels, one or two percentage points around 95% precision are statistically indistinguishable.
- Scoring and miss adjudication were performed by an LLM adjudicator, not a human domain expert.
The 92.5% and 96.5% above were measured in July 2026 against a curated golden set of 173 items. Building the English measurement as an exhaustive enumeration — every document, down to table rows — gave us a harsher yardstick, so we applied it to Korean too: an exhaustive key of 2,013 items across the same three solicitations, about 12× the old denominator (671 items per notice). The key is frozen; 28 labels later proved invalid and were removed, so the scored denominator is 1,985 items.
On that yardstick the first measurement was 73.5%. We published the low number, then made the fix its miss classification pointed to — extracting each instruction sentence in a statement of work as its own requirement instead of collapsing a paragraph into one — and re-scored against the same frozen key: 87.4%, and 92.0% on critical items. Misses fell from 525 to 250.
We re-checked the legacy 173-item yardstick with the improved extractor as well: 95.4% on token match alone — no regression, it improved too. The headline figure stays the exhaustive 87.4%, because that is the yardstick the English number is measured on, and because a harsher denominator is the honest one to lead with.
English (U.S. federal · SAM.gov) measurement
English solicitations, measured the same way — August 2026.
The "formal English golden set in preparation" status is over. We measured against an exhaustive golden set built from all 31 documents of 4 currently-posted SAM.gov solicitations (attachments, amendments, pricing matrices and a wage determination included), under the same blind protocol as the Korean set — the answer key was frozen before the extractor ran.
- 93.2%
- Requirement extraction recall — exhaustive golden set, 1,481 items (4 solicitations, all documents). Journey on the same frozen yardstick: 85.3% → 91.9% → 93.2% across two improvement rounds
- 97.0%
- Critical-item recall — 888 items whose omission is disqualification-grade (91.7% before the improvement rounds)
- 14 / 19
- Planted defects caught in the English adversarial judgment benchmark — 11 of them machine-certain
- 103 / 103
- Machine-certain judgment flags all correct in the three-vote unanimous run (2026-08-04) — zero misfires; the earlier two-vote run scored 112 / 112 after amendment-supersede handling was added
This 93.2% and the Korean 87.4% are scores on the same exam — both golden sets enumerate every document exhaustively (370 items per solicitation in English, 671 in Korean). English is ahead today because it has been through two improvement rounds on this yardstick and Korean has just finished its first. The 92.5% we used to publish for Korea was scored on the easier curated yardstick, so we do not place it next to this figure.
Publishing the miss classification worked exactly as intended — twice. Round one fixed the clause-reference rule (misses 218 → 120), round two fixed option-period pricing rows (120 → 101; pricing rows 13 → 3). The remaining 101 are classified and published too: 67 scattered content requirements — a long tail with no single rule left to fix — plus 25 form/submission and 9 others. Grounding rate: 97.0–99.3%.
Judgment was measured on a real USFK solicitation (sterilizer maintenance, 51st Medical Group, Osan AB) with 19 planted defects across 8 synthetic response packs — an exam whose answer key is fixed by construction. 11 of the 14 catches were machine-certain; of the 6 hardest evasion-grade defects, 3 were caught.
The 5 defects judgment missed (each accounted for)
- A grand total written out in words ("one hundred forty-one thousand …") — the price consistency check parses numeric notation only
- A "5 business days" softener added in parentheses after the 72-hour reporting commitment
- Scope quietly redefined to PM only on "units in clinical use"
- SAM registration described as "renewal in progress"
- Invoicing system mix-up (IPP vs WAWF) — a known gap: that clause routes to performance conditions, outside offer matching
- English judgment precision: in the latest benchmark run, all 103 machine-certain flags were correct — zero misfires, with defect recall at 14/19. We also measured the wobble: judging the same input repeatedly moves the detection count (standard deviation 0.69 on the two-vote configuration). So judgment now takes three votes instead of two, still requiring unanimity — switching to a majority rule would raise the number of certain flags but raise misfires with it, which would eat the very precision this page is about. Whether the wobble actually shrank is a figure that only appears across many runs, so what we can say today is that recall and misfires did not get worse after the change.
- The answer key was labeled exhaustively by an independent prompt and model, merged and audited by the adjudicator, then frozen; every miss was individually adjudicated. The full procedure is in the measurement document in the repository.
- These are benchmark figures, not a guarantee on your actual documents.
The reports the pipeline produces are public too — read one before sending anything.