OpenAddresses real-point resolver eval — the non-circular accuracy track (2026-05-30)
Direction-C resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-depth. The first non-circular end-to-end accuracyharness pass rateWhether a full address parses end-to-end with every component correct and well-segmented and no orphan spans. Stricter than label F1 — a model can win on recall yet fail the harness if boundaries slip. number
for the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. — real US addresses with real government coordinates, resolved
against a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. they don't come from — and, because v0 is a TypeScript port
of the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser, the neural-vs-PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-parser headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. at the same
time (the parser axis of "beat PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.", with no Docker PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. stack needed).
Why this is the honest scoreboard
The WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-bootstrap evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. (the +8.5pp exact-match-tiering result) renders WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. places back into address strings and resolves WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.→WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.. It's a legitimate ranking test, but it's circular by construction — the ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy. is the same gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. consults, so it can't measure whether we resolve real-world addresses to the right place on the map.
OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. is independent: each row is a real US address with a real lat/lon harvested from authoritative government address pointssitus dataA dataset of exact address-point coordinates (rooftop-level). Mailwoman's geocoder uses a national situs layer (124.9M US points built from state address-point sources) as the highest-precision tier of the geocode cascade., and the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. consults the WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. — a different source. So:
- Admin-match (did we resolve to the expected localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., by canonical gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. name vs OA's ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy.) measures resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. correctness independent of WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. id conventions.
- Coordinate error (great-circle from the resolved admin centroid to OA's real point) is a genuine map-accuracy signal — un-gameable, since OA's point was never in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture..
The set: data/eval/external/openaddresses-us-sample.jsonl (10,000 rows, 8
statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., stratified dense-urban → rural so no single stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. dominates).
Two-tier metric
Per the DeepSeek resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. consult, a sub-10km coordinate bar is impossible for admin-centroid resolution — a city centroidlocality centroidThe representative centre point of a city or locality, used as a coarse coordinate when no exact address point is available — the coarsest tier of the geocode cascade. is legitimately tens of km from its edge addresses. So the metric is split:
- Admin-match Acc (the headline) — localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match and regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match rates, granularity-independent.
- Coord error p50/p90 — reported as the admin-centroid tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous.. The streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-level tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. (TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data.) will own a sub-km coordinate bar in a later phasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary)..
Head-to-head: neural vs the Pelias parser (v0.7.2 model, 10,000 rows)
mailwoman's v0 rule parser is a TypeScript port of the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser, so
running both parsers through the same resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. makes this a direct
neural-vs-PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-parser comparison on real, non-circular addresses — no Docker
PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. stack required. The table below is emitted verbatim by the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. runner
(--out-md); evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. figures are never hand-typed (see the integrity note).
| parser | localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match | regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match | resolved | coord p50 km | coord p90 km | p99 km |
|---|---|---|---|---|---|---|
| neural | 96.1% | 100.0% | 100.0% | 2.4 | 10.6 | 25.0 |
| v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.) | 94.4% | 99.5% | 99.8% | 2.4 | 10.6 | 25.0 |
Neural beats the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser on real US addresses — +1.7pp localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., +0.5pp regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., and a higher resolve rate — and wins in every stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. (per-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. below). Both share identical coordinate error because they feed the same resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. and, when both resolve to the right admin, land on the same centroid; the difference is purely which addresses each parser resolves correctly at all.
Neural per-state (locality-match)
| stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. | n | neural loc | v0 loc | neural reg | v0 reg |
|---|---|---|---|---|---|
| CA | 1429 | 99.9% | 99.7% | 100.0% | 99.9% |
| DC | 1429 | 99.5% | 99.2% | 99.9% | 99.2% |
| IA | 1429 | 94.3% | 86.4% | 99.8% | 99.0% |
| IL | 1429 | 98.7% | 97.6% | 100.0% | 99.7% |
| MT | 1428 | 96.7% | 95.3% | 100.0% | 99.4% |
| SD | 1428 | 96.8% | 96.8% | 100.0% | 99.7% |
| VT | 1428 | 87.1% | 85.7% | 100.0% | 99.5% |
Headline: neural localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match 96.1%, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match 100.0% on 10,000 real US addresses, resolved 100.0%; coord p50 2.4km / p90 10.6km / p99 25.0km (admin-centroid tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. — median is centroid-to-address distance, not a geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. miss). Neural's largest margin over the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser is IA +7.9pp (suburban/ rural midwest); the weakest stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. for both is VT (rural-northeast, sparse gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.), where neural still leads 87.1% vs 85.7%.
Eval-integrity note
This doc's tables are produced by scripts/eval/oa-resolver-eval.ts --out-md and
pasted verbatim. The runner also writes --out-json; the two are computed from the
same aggregates so they cannot disagree. (Earlier in this work an OA table was
hand-typed and shipped wrong numbers — the self-reporting --out-md flag exists to
make that class of error impossible.)
What it measures vs. doesn't
- It does confirm the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. maps real addresses to the right citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. at scale, independent of the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.'s own id scheme — the credibility check the WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-bootstrap number couldn't give.
- It does not measure streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./house precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. (the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. is admin-level; coord error reflects centroid-to-point distance, not a geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. miss).
- RegionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match required a name↔abbrev map: resolved regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. carry the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.'s canonical full name ("California", "District of Columbia") while OA carries the USPS abbreviation ("CA", "DC"). An early cut scored regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match at 30% purely from that mismatch — a matcher bug, not a resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. one; fixed in the runner.
Resolver change that landed with this
core/resolver/resolve.ts now stamps metadata.resolver_name (the resolved
place's canonical gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. name) alongside resolver_score. Without it the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.
could only compare against the parser's own text spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree., not the place the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.
actually chose — so it couldn't tell a right-name/wrong-place resolution from a
correct one. The name is also generally useful to consumers (display the canonical
name, not the raw input spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.).
Reproduce
node scripts/eval/oa-resolver-eval.ts \
--eval data/eval/external/openaddresses-us-sample.jsonl \
--model <v0.7.2.onnx> --tokenizer <v0.6.0-a0> --model-card <card> \
--wof admin-global-priority.db,postalcode-us.db \
--out-json /tmp/oa-full.json
--limit N for a quick subset. Per-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. breakdown is in the runner's output.