Direction C — Phase 1 end-to-end resolver eval (2026-05-30)
The first "address string → correct WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. place" benchmark, and the kill/continue gate for the per-input routing thesis. Headline: the win was fixing the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon., not routing parsers.
Setup
- Ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy.:
scripts/eval/gen-wof-bootstrap.pysamples real US WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. localities (2,406 rows, 401 localities × 51 regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. × canonical/lowercase/nocomma), rendered to address strings, labelled with the source WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. id + centroid (hierarchy-tolerant: localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. or its regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. accepted). Sampled from the custom gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (never the off-the-shelf dumps — different ids). - Runner:
scripts/eval/resolver-eval.tsparsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. each input two ways — neural (v0.7.2) and v0-via-adapter (scripts/eval/v0-tree-adapter.ts, flat record → tree) — resolves both once through the shared WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon., and derives all baselines from the two resolutions (neural-only / v0-via-adapter / arbiter=pick-higher-resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-score / oracle=correct-if-either). Metrics: hierarchy-tolerant Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. + great-circle error.
The resolver was broken on realistic US addresses — and we fixed it
The first run exposed two resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. gaps the deleted off-the-shelf DB had masked:
- No top-level countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. constraint —
ResolveOptsonly propagated countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. down from a resolved parent, so a bare "IL" over the 7-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. fuzzy-matched a French regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. (@48.15,−1.64), poisoning the whole walk. - US stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviations didn't resolve — WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. carry "Illinois", not "IL";
findPlace('IL')returned nothing, killing the parent-constraint the walk depends on.
Fixes (both production keepers):
ResolveOpts.defaultCountry— a top-level countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. hint (set from the localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.-gate).- RegionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-abbreviation enrichment —
scripts/add-region-abbrevs.tsadds each regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.'s abbreviation as a searchable name, sourced from the in-repo chromium-i18n / libaddressinput dataset (sub_keys↔sub_names), 51 abbrevs across 7 countries. (TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data.'scorpus/src/codex/us-fips-state.tsis the US-specific alternative.) - Plus
resolver_scorestamped on resolved nodes (for the arbiter + downstream).
| metric | before fixes | after |
|---|---|---|
| Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. (all) | ~10% | 68.9% |
| coord error p50 | 843 km | 0.0 km |
| coord error p90 | 7,640 km | 1,090 km |
Kill/continue gate (full 2,406-row run)
| baseline | clean | perturbed | all |
|---|---|---|---|
| neural-only | 77.1% | 64.8% | 68.9% |
| v0-via-adapter | 69.5% | 60.8% | 63.7% |
| arbiter | 76.9% | 69.5% | 72.0% |
| oracle | 79.4% | 77.1% | 77.9% |
- Arbiter beats neural-only by +3.1pp overall and +4.7pp on noisy input. Gate: all +3.1 (≥3 ✓), perturbed +4.7 (≥−2 ✓), clean −0.1 (≥5 ✗).
- The strict clean-5pp criterion fails on an inverted premise: it assumed v0 wins clean (so the arbiter would pick v0's clean wins). End-to-end, neural wins clean (77% vs v0's 70%) — the opposite of the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.-level capability map (libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it.). The routing gains correctly land on noisy + overall, where the robustness story lives.
- Oracle shows ~6pp of headroom above the arbiter — a better router (lexical quality signal, not just resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. score) could capture more.
Conclusions
- The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. was the bottleneck, not parser routing. Fixing countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. + abbreviation coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. took the stack from broken (~10%) to 68.9% Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. / 0 km median. Ship neural→resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. as the geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. stack.
- Routing is a real but modest lift (+3.1pp overall, +4.7pp noisy). Worth building as the banded version (cheap lexical router → arbiter only on the noisy/ambiguous band), so the dual-parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. cost is spent only where it pays and clean inputs stay on neural.
- The biggest remaining lever is resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present., not the parser: more aliases/ abbreviations, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-level resolution (TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data. — WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. has no streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. node), more localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. The p90 of 1,090 km is the cross-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. ambiguity tail when a regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. fails to constrain.
Caveats: synthetic WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-rendered addresses (the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. track —
data/eval/external/openaddresses-us-sample.jsonl, 10k real US points — is the
independent coordinate-error check, to run next); admin-level only (no streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./house).