US source-independent holdout — first numbers (2026-06-10, night-10)

The re-scoped #472: 6,453 rows across 44 statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. whose Overture provenance chain contains zero OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries.-derived datasets (100% DoT NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses.) — rows our trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. lineage has never carried. This is the memorizationoverfittingWhen a model memorizes quirks of its training set instead of learning general patterns, so it scores well in training but poorly on new data. Guarded against with held-out evals and regularization. axis, orthogonal to the VT/WY/ND geographic holdout (which is geography-independent but lineage-shared). Builder: scripts/eval/build-nad-holdout.ts (deterministic hash sampling, seed 42, release 2026-05-20.0); render: <number> <street>, <city>, <state> <postcode>.

Results (admin-centroid tier; admin-match is the headline, coord secondary)

parser	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match	regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match	resolved	coord p50	coord p90
v4.1.0	78.1%	99.9%	100.0%	5.4 km	106.7 km
v4.2.0	77.3%	100.0%	100.0%	5.5 km	138.8 km
v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. rules)	86.7%	91.7%	99.4%	4.3 km	24.6 km

Finding 1 — no memorization cliff

v4.2.0 vs v4.1.0 on never-seen-lineage data: localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. −0.8, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. +0.1 — the same within-noise story as the lineage-shared evalsevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.. The consolidation's gains are not riding memorized OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. rows. This was the question this holdout exists to answer, and the answer is clean.

Finding 2 — the census-designation gap (CORRECTED same night)

CORRECTION (night-10, hours after first publication): the first version of this section attributed the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. gap to the postal-citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./vanity-citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. (#475). The alias-table join MEASURED that attribution and refuted it: only 1 of 461 named misses is alias-explained — NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses.'s localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. field carries census/municipal names, not postal names, so this evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. cannot exhibit the vanity-citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. failure mode at all.

What the misses actually are (classified, n=461 with a resolved-but-wrong name):

• 54.0% are the SAME PLACE under a different name surface — census designations: College CDP ↔ WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. Fox Farm-College, City and Borough of Juneau, X Township ↔ X (NJ alone is 148 of 461 — townships). This is the Plauen-Vogtl name-match-artifact class, US edition; the fix lane is the #386-style hierarchy-aware designation credit generalized to US census surfaces — filed as its own issue with these numbers.
• The remainder are genuine ranking/disambiguation misses (e.g. Juneau → Wrangell).

Consumers: the designation-credit issue (primary); #478 still gets its complementarity row (v0 localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. 86.7 vs neural regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. 100.0). #475's alias table is NOT validated by this evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. — it needs an evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. whose inputs carry postal surfaces (real-traffic shaped); noted on the issue.

Caveats: admin-centroid coord errors are expected to be tens of km (the harness's own note); v0's 24.6 p90 reflects its conservative no-resolve behavior on hard rows (99.4% resolved). The NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. render is one template — format diversity is not this evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.'s job.

Trust: 6,453 rows ≥ the 1000-row floor → TRUSTED. Files: data/eval/external/overture-us-nad-holdout.jsonl + .report.json.

Morning-after correction №2 (2026-06-10 day): the gold itself was composite

The night's Finding-2 classification was computed on broken gold: NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses.'s localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. field sometimes carries a composite "County, Municipality" string (New Jersey systematically — "Union, Union Township", "Cape May, Sea Isle City"), and the holdout builder graded against the whole composite. Correct resolves failed on the county prefix; NJ's 148 "misses" were ~all this artifact. Builder fixed (last comma segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context.), holdout rebuilt.

Corrected numbers (v4.2.0): localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match 78.5 (was 77.3), v0 87.7 (was 86.7) — the fix is fair to both. Misses-with-candidate drop 461 → 317, NJ leaves the top-miss statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. entirely, and the strict designation class (<name> <designation> ↔ <name>) shrinks to 14 rows (~0.2pp) — far below implementation threshold. #498 closed as premise-refuted-by-corrected-measurement. The real residual gap (9.2pp vs v0) is ranking/ coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. misses concentrated in ID/WY/UT/LA plus WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. neighborhood-vs-localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. granularity (College CDP ↔ Fox Farm-College) — the latter is the PIP lens's territory, not a name-credit's.

Running tally for this document: three findings published, two corrected by their own follow-up measurements, within 18 hours. The instrument is part of the experiment.