measurement, published

We publish how the numbers were made.

This page discloses how we measured on Korean public procurement (나라장터) notices. What was measured, how, what the pipeline missed — enumerated one by one — and where the measurement itself falls short.

87.4%
Korea (나라장터) — requirement-extraction recall, frozen exhaustive golden set of 2,013 items (1,985 valid after removing 28 invalid labels)
96.5%
Korea — machine-certain judgment precision, after adjudicating every flagged case
99.4%
HWP·PDF table preservation — 500 attachments from one day of 나라장터 notices (2026-09-08), preferred format, parse errors excluded
How the measurement is run

The answer key is authored exhaustively and independently, with no access to the pipeline output, and every item carries a verbatim anchor that is machine-verified against the source. The key is frozen in a commit before the extractor runs, and after scoring, every item the pipeline missed is adjudicated one by one — nothing is left as a mystery.

When we change the extractor, we re-score against that same frozen key. So the before-and-after figures on this page are scores on the same exam, not a new exam with a friendlier denominator.

Korean public procurement (나라장터)

The golden set

Two real, published negotiated-contract service tenders (8 documents, table-preserving parse). An adjudicator authored the full requirement set — 85 and 88 items, 41 of them critical — while barred from seeing any pipeline output, so the answer key could not be shaped by what the pipeline happens to extract.

Every item carries a verbatim anchor into the source text, and all 173 anchors pass machine verification. Table rows are included as items — tables are where Korean tender documents put the substance.

Measuring extraction

A single extraction pass scores 82.1% recall against the golden set. Adding one critic pass — a second call that only hunts for requirements the first pass missed — lifts it to 92.5%, beating an equal-cost union of two independent passes (88.4%). That is the configuration we ship: one extra LLM call per document.

The critic over-extracts by design; the deterministic grounding layer absorbs it — every added item, grounded rate 100%.

Measuring judgment

Real bid submissions are exempt from disclosure in Korea, so they cannot be obtained. Instead: for three real published tenders (engineering design, solar installation works, security services — HWP and PDF), we generated fully compliant bid packages, then injected 45 defects with seeded, fixed mutations — so every defect label is certain — plus 15 harmless-rewording traps.

The 18 / 19 figure on the verification page comes from a separate test — the adversarial corpus: packages seeded with 19 subtle defect types plus 2 clean packs for false-positive measurement, re-run whenever the prompt or model changes. The latest Korean run (2026-08-06) caught 18 of the 19, 17 of them machine-certain. It uses a different corpus and defect set from the 45 injected defects below, so the two numbers are not comparable with each other.

Machine-certain precision: 96.5% after adjudicating every flagged case. The deterministic layer caught every missing document, every identifier tamper, and every total-amount tamper — without LLM involvement.

What it missed

Every miss was adjudicated individually — none are mysteries. Of 45 injected defects, the judgment layer missed 9:

Also disclosed: 2 of the 15 harmless-rewording traps drew a “partial” flag (false positives), and the extraction residue clusters in edge cases such as authoring instructions inside a proposal’s table-of-contents table.

Stability

All 15 packages were verified twice, independently. The final pass/block posture agreed 15/15. What flickers between runs is confined to borderline grades — “is this commitment strong enough” — never the defect-anchored findings (missing documents, identifier and total mismatches, dropped requirements).

What is different — generic tooling vs. this engine

For partners and auditors: the comparison is against generic document tooling and profile-vs-criteria checkers, not against any named vendor.

Table extraction from HWP/PDFdrops 85–87% of table cells (measured on 214 notices)preserves 99.4% of tables (500 attachments, one day)
What is judgedcompany profile vs. scoring criteriasubmitted documents vs. each requirement in the notice
Confidencea single model answerthe same question asked three times, all answers agreeing + verbatim grounding; anything less is flagged for review
Before delivery—seven automated gates before delivery
Not measured here—evaluation scoring, performance interpretation, real vendor proposals (packages are synthetic)

Numbers in this table use the same standards as the sections above; each has its own measurement page or note. Reproduce the tokenizer measurement (public repository) →

Limits of the measurement
Korea, re-measured — August 2026

The 92.5% and 96.5% above were measured in July 2026 against a curated golden set of 173 items. We then built a harsher yardstick — every document enumerated exhaustively, down to table rows — and measured again: an exhaustive key of 2,013 items across the same three solicitations, about 12× the old denominator (671 items per notice). The key is frozen; 28 labels later proved invalid and were removed, so the scored denominator is 1,985 items.

On that yardstick the first measurement was 73.5%. We published the low number, then made the fix its miss classification pointed to — extracting each instruction sentence in a statement of work as its own requirement instead of collapsing a paragraph into one — and re-scored against the same frozen key: 87.4%, and 92.0% on critical items. Misses fell from 525 to 250.

We re-checked the legacy 173-item yardstick with the improved extractor as well: 95.4% on token match alone — no regression, it improved too. The headline figure stays the exhaustive 87.4%, because a harsher denominator is the honest one to lead with.

The reports the pipeline produces are public too — read one before sending anything.