We publish how the numbers were made.
This page discloses how we measured on Korean public procurement (나라장터) notices. What was measured, how, what the pipeline missed — enumerated one by one — and where the measurement itself falls short.
- 87.4%
- Korea (나라장터) — requirement-extraction recall, frozen exhaustive golden set of 2,013 items (1,985 valid after removing 28 invalid labels)
- 96.5%
- Korea — machine-certain judgment precision, after adjudicating every flagged case
- 99.4%
- HWP·PDF table preservation — 500 attachments from one day of 나라장터 notices (2026-09-08), preferred format, parse errors excluded
The answer key is authored exhaustively and independently, with no access to the pipeline output, and every item carries a verbatim anchor that is machine-verified against the source. The key is frozen in a commit before the extractor runs, and after scoring, every item the pipeline missed is adjudicated one by one — nothing is left as a mystery.
When we change the extractor, we re-score against that same frozen key. So the before-and-after figures on this page are scores on the same exam, not a new exam with a friendlier denominator.
Korean public procurement (나라장터)
Two real, published negotiated-contract service tenders (8 documents, table-preserving parse). An adjudicator authored the full requirement set — 85 and 88 items, 41 of them critical — while barred from seeing any pipeline output, so the answer key could not be shaped by what the pipeline happens to extract.
Every item carries a verbatim anchor into the source text, and all 173 anchors pass machine verification. Table rows are included as items — tables are where Korean tender documents put the substance.
A single extraction pass scores 82.1% recall against the golden set. Adding one critic pass — a second call that only hunts for requirements the first pass missed — lifts it to 92.5%, beating an equal-cost union of two independent passes (88.4%). That is the configuration we ship: one extra LLM call per document.
The critic over-extracts by design; the deterministic grounding layer absorbs it — every added item, grounded rate 100%.
Real bid submissions are exempt from disclosure in Korea, so they cannot be obtained. Instead: for three real published tenders (engineering design, solar installation works, security services — HWP and PDF), we generated fully compliant bid packages, then injected 45 defects with seeded, fixed mutations — so every defect label is certain — plus 15 harmless-rewording traps.
The 18 / 19 figure on the verification page comes from a separate test — the adversarial corpus: packages seeded with 19 subtle defect types plus 2 clean packs for false-positive measurement, re-run whenever the prompt or model changes. The latest Korean run (2026-08-06) caught 18 of the 19, 17 of them machine-certain. It uses a different corpus and defect set from the 45 injected defects below, so the two numbers are not comparable with each other.
Machine-certain precision: 96.5% after adjudicating every flagged case. The deterministic layer caught every missing document, every identifier tamper, and every total-amount tamper — without LLM involvement.
Every miss was adjudicated individually — none are mysteries. Of 45 injected defects, the judgment layer missed 9:
- 3 — the notice figure lived in a table the (then-current) parser dropped; the table-preserving parse path since became the default
- 2 — the extracted requirement summary omitted the number needed to catch the deviation (granularity, not judgment)
- 2 — genuinely lenient judgment calls: a deleted section still voted “covered”. These are the real judgment failures: 2 of 45.
- 1 — an upstream extraction miss propagated downstream
- 1 — label turned out invalid: the deleted content was covered by another document
Also disclosed: 2 of the 15 harmless-rewording traps drew a “partial” flag (false positives), and the extraction residue clusters in edge cases such as authoring instructions inside a proposal’s table-of-contents table.
All 15 packages were verified twice, independently. The final pass/block posture agreed 15/15. What flickers between runs is confined to borderline grades — “is this commitment strong enough” — never the defect-anchored findings (missing documents, identifier and total mismatches, dropped requirements).
For partners and auditors: the comparison is against generic document tooling and profile-vs-criteria checkers, not against any named vendor.
| Table extraction from HWP/PDF | drops 85–87% of table cells (measured on 214 notices) | preserves 99.4% of tables (500 attachments, one day) |
|---|---|---|
| What is judged | company profile vs. scoring criteria | submitted documents vs. each requirement in the notice |
| Confidence | a single model answer | the same question asked three times, all answers agreeing + verbatim grounding; anything less is flagged for review |
| Before delivery | — | seven automated gates before delivery |
| Not measured here | — | evaluation scoring, performance interpretation, real vendor proposals (packages are synthetic) |
Numbers in this table use the same standards as the sections above; each has its own measurement page or note. Reproduce the tokenizer measurement (public repository) →
- The bid packages are synthetic. Real vendors’ proposals differ in style, structure, and error distribution — figures here are benchmark performance, not a guarantee on your documents.
- Judgment-of-merit factors — evaluation scoring, how track-record rules are interpreted — are out of scope; only document, figure, and commitment-level checks were measured.
- With 45 defect labels, one or two percentage points around 95% precision are statistically indistinguishable.
- Scoring and miss adjudication were performed by an LLM adjudicator, not a human domain expert.
The 92.5% and 96.5% above were measured in July 2026 against a curated golden set of 173 items. We then built a harsher yardstick — every document enumerated exhaustively, down to table rows — and measured again: an exhaustive key of 2,013 items across the same three solicitations, about 12× the old denominator (671 items per notice). The key is frozen; 28 labels later proved invalid and were removed, so the scored denominator is 1,985 items.
On that yardstick the first measurement was 73.5%. We published the low number, then made the fix its miss classification pointed to — extracting each instruction sentence in a statement of work as its own requirement instead of collapsing a paragraph into one — and re-scored against the same frozen key: 87.4%, and 92.0% on critical items. Misses fell from 525 to 250.
We re-checked the legacy 173-item yardstick with the improved extractor as well: 95.4% on token match alone — no regression, it improved too. The headline figure stays the exhaustive 87.4%, because a harsher denominator is the honest one to lead with.
The reports the pipeline produces are public too — read one before sending anything.