The honest eval — when the metric graded by name and missed the wrong state
2026-06-08. Self-emitted figures from scripts/eval/honest-eval.sh (#371 leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits.-free
geographic split + #373 PIP-containment). Numbers are written by the harness, never hand-typed.
Why a new yardstick
Every resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. number we have ever quoted shares a flaw: the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set and the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. cover the same places. A modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that has seen Springfield, Illinois a thousand times in trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. will "resolve" it on the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. — but that is recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. of a memorized place, not generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure.. And the headline metric made it worse. We graded resolution by localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. name-match: did the resolved place carry the same name string as the gold? That question is blind to the failure that actually matters — resolving the right name in the wrong place. "New York" matches "New York" whether the point lands in Manhattan or in a village 280km upstate.
So this harness measures two things the old one could not:
- A leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits.-free slice. Evaluate only on OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. rows whose geography the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.
corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. held out (
corpus/src/split.tsdefaultHoldouts(): US = VT/WY/ND, FR = Corse/Lozère/ Creuse). Those are places the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' has never seen. In the current samples only US/Vermont clears a 1000-row trust floor (1428 rows); FR's held-out départements total 16 rows, and DE has no manifest holdout. We report VT and flag the rest honestly rather than scoring noise. - A non-gameable metric. Beyond name-match, report regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match, coordinate error (great-circle, gold point to resolved centroid, p50/p90), and PIP-containment — is the gold point inside the resolved WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. polygon. PIP is name-surface-independent: it rewards a geographically correct resolve even when WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.'s canonical name differs from the gold's.
What it found, immediately
The first run on the honest Vermont slice:
| metric | value |
|---|---|
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. name-match (the old, gameable number) | 93.7% |
| regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match | 0.0% |
| coord p50 / p90 (km) | 326 / 1827 |
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. PIP-containment | 11.3% |
A 93.7% name-match next to a 326km median coordinate error is the whole argument for this harness in one line. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. was finding the right name and the wrong place nearly nine times in ten, and the metric we had been quoting could not see it.
Root cause
The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is fine. It tags region="VT", locality="North Hero", the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., the number, and the
postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. cleanly. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. is where it breaks: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. stores the stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. only as "Vermont", and the
place_search FTS index carried no USPS abbreviations, so findPlace("VT") returned nothing. With
no resolved regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.'s parent-constraint never engaged, the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. search ran
unconstrained across the entire US, and a higher-population namesake in another stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. won. "Sheldon"
has ten US localities; the population-5,455 one beats Vermont's population-932 one every time.
The abbreviation enrichment that would fix this already exists — scripts/add-region-abbrevs.ts
sources stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality./provinceregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviations from the in-repo chromium-i18n libaddressinput data and writes
them into the names table, which build-fts folds into place_search. It was simply missing from
the build manifest's post-build step, so the deployed gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. was built without it (0 abbr rows).
The fix, measured on the same slice
Running add-region-abbrevs.ts and rebuilding the FTS index, then re-measuring on the identical VT
slice:
| metric | baseline | abbrev-enriched |
|---|---|---|
| regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match | 0.0% | 99.9% |
| coord p50 (km) | 326.3 | 3.4 |
| coord p90 (km) | 1826.7 | 7.4 |
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. name-match | 93.7% | 93.7% |
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. PIP (all rows) | 11.3% | 29.7% |
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. PIP (polygon-coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.-adjusted) | 15.1% | 47.1% |
RegionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. resolution goes from broken to near-perfect; the median coordinate error collapses from 326km to 3.4km. Name-match does not move — confirming the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' was always emitting the right name; the fix makes the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. pick the right instance.
Note the PIP-containment numbers. Even after the fix, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-PIP is 29.7%, because 37% of the correctly-resolved Vermont localities are WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. point geometry (no polygon can contain a point), and WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.'s small-town polygons are tight while OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. ascribes rural addresses to the town they are nearest, not the town whose boundary encloses them. That is why the scorecard leads with regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match and coordinate error — both checkable for 100% of rows — and reports localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-PIP only alongside its polygon-coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. denominator. Raw localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-PIP would silently count un-PIP-able points as failures.
The catch — and why functional checks come before verdicts
The fix is not a clean win, and the demo presets caught why. On four populous US presets the abbreviation enrichment moved regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match from 25% to 100% but regressed localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. name-match from 100% to 75%, with one preset's coordinate error jumping to 283km:
350 5th Ave, New York, NY 10118— baseline resolves "New York" (NYC); abbrev-enriched resolves "New York Mills", a village 283km upstate.
The mechanism is a second, deeper data gap. Once the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. resolves, the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. boosts candidates
that descend from it. New York Mills carries a full ancestry chain in WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. (localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. → localadmin →
county → New York stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. → US). New York CitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.'s ancestors row contains only itself — its chain
to New York stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. is missing. So the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-descendant boost lifts the village and not the citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., and
the village's boosted score overcomes the citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.'s population. The regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-abbreviation fix is
netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network.-positive for well-parented rural places and netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network.-negative for mis-parented metros.
The lesson the house already knew, re-earned: aggregate metrics agreed the fix was good (Vermont went from 326km to 3.4km); the functional presets disagreed (NYC broke). When they disagree, the functional check wins — and chasing the disagreement found the deeper bug.
The resolution — repair ancestry from wof:hierarchy
The ancestry gap is a build artifact, not a source gap. NYC's source geojson carries a full
wof:hierarchy (region_id 85688543 = New York in every one of its five boroughboroughAn administrative or historical division of a city — e.g. the five boroughs of New York City. May be postal, legal, or both, and complicates the locality hierarchy. branches); it is
only wof:parent_id that is -4, and build-unified-wof's parent_id-closure follows nothing but
parent_id. So scripts/backfill-ancestors-from-hierarchy.ts reads wof:hierarchy for every
only-self place and inserts the missing ancestor rows — repairing 47,129 places (+132,832 rows) in
the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture..
Re-measured on the abbreviation-enriched and ancestry-backfilled gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.:
| slice | metric | baseline | abbrev only | abbrev + backfill |
|---|---|---|---|---|
| VT held-out (1428) | regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match | 0.0% | 99.9% | 99.9% |
| VT held-out | coord p50 / p90 (km) | 326 / 1827 | 3.4 / 7.4 | 3.4 / 7.4 |
| full-US (10k) | regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match | 14.2% | 99.9% | 99.9% |
| full-US (10k) | coord p50 / p90 (km) | 6.5 / 2763 | 3.3 / 10.3 | 3.3 / 10.3 |
| demo presets (4) | localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match | 100% | 75% ⚠ | 100% ✓ |
| demo presets | NYC resolves to | NYC | New York Mills ✗ | NYC ✓ |
The wrong-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. error tail (full-US coord p90) collapses from 2763km to 10.3km, and the metro regression is gone, with NYC resolving to New York CitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. again. Netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network.-positive on rural and metro.
Two things are worth saying about what this fix is. It is parser-agnostic: the harness runs both the neural parser and the v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-style rules) parser through the same resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon., and the fix lifts both from 0% regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. / ~330km coord to ~99.5% / ~3.3km. The bug lived in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., not the parser. And once the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. is honest, the neural parser keeps its edge over v0 on geography it never trained on:
| parser (VT, fixed DB) | localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. | regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. | coord p50 | coord p90 | coord p99 |
|---|---|---|---|---|---|
| neural | 93.7% | 99.9% | 3.4 | 7.4 | 277 |
| v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.) | 91.9% | 99.5% | 3.3 | 7.2 | 2120 |
The medians tie; the neural parser's p99 tail (277km vs 2120km) is far tighter, which is the beat-PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-on-the-hard-cases result we want on a held-out slice.
Status
- The harness (
scripts/eval/honest-eval.sh) + coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.-adjusted PIP reporter (scripts/eval/pip-containment.py) are the yardstick. - The fix is two idempotent build steps —
scripts/add-region-abbrevs.ts(already existed; was absent from the manifest) andscripts/backfill-ancestors-from-hierarchy.ts(new) — now wired intoscripts/wof-build-manifest.json's post-build, before FTS. - The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. fix is validated on a copy (
admin-abbrev-test.db) but not promoted to the canonical DB or the live demo — that swap + R2 re-publish is the operator's call (a one-shot: run the two steps onadmin-global-priority.db, rebuild FTS, rebuild the slimwof-hot.db, re-publish). The canonical DB is untouched.
Next
- Promote: run the two build steps on the canonical gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., rebuild FTS + the slim demo DB, re-publish to R2 (smoke-test regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match ≥99.9%, coord p50 ≤5km after).
- Fold the ancestry repair into
build-unified-wof'spopulateAncestorsso a fresh build is correct without the post-build step. - Broaden the honest slice: a targeted OA re-ingest for FR's held-out départements, and a DE holdout in the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. manifest, so regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match and coordinate error can be reported per localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for..