Resolver failure analysis (v0.7.2, 1500 OA real points)

Read-only triage of where the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. misses, run while the v0.8.1 MLM arms trained. Method: scripts/eval/oa-resolver-eval.ts --limit 1500 --errors-json (new flag, this branch) scores the real v0.7.2 neural parser AND v0/PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. through the same WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. on real OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. points, and dumps every row where either parser missed localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., carrying each parser's resolved admin name so a miss can be bucketed as resolve-wrong vs unresolved.

Headline

parser	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-match	regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match	resolved
neural	94.9%	99.9%	100.0%
v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.)	93.7%	99.4%	99.7%

Neural beats v0/PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. on real points (+1.2pp localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.) — the north-star claim, on an honest non-circular signal. 101 rows had a miss from one or both parsers.

Where neural's 76 locality-misses actually go

bucket	count	what it is
(A) name-form variantname-form variantA resolver 'miss' that is really a measurement artifact: the right place is found but its gazetteer-canonical name differs from the eval's expected literal — 'St. Johnsbury' vs 'Saint Johnsbury'.	40	resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. found the RIGHT place; the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.'s name-matcher rejects the form
(B) wrong place	13	resolved to a real but incorrect admin — the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-disambiguation target
(C) no localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. resolved	23	parser mis-segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context., <Null> tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., or WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. gap

If bucket (A) were credited, neural localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. on these 1500 rows is ≈97.6%, not 94.9%. And several of (B) are also name-form (Mt Pleasant→Mount Pleasant, Mc Laughlin→ McLaughlin, Enosburgh→Enosburg Center), so true localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. accuracy is ~97–98%.

The evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. has been underselling the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.. Examples from (A): Butte→Butte-Silver Bow (consolidated citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-county — WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.'s canonical name), Saint Johnsbury→St. Johnsbury, Derby→Derby Center, Monroe Twp→Monroe, Barre City→Barre.

Recommended next PRs, ranked by ROI

1. EvalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. name-match fidelity (HIGH ROI) — ✅ SHIPPED IN THIS PR. Match OA's expected name against the resolved place's full WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. names altname set (normalized), not a single canonical string, plus a tight abbreviation layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. (St→Saint, Mt→Mount, Ft→Fort, Mc-despace, diacriticsdiacriticAn accent mark that modifies a letter (é, ñ, ç). Address normalization must fold diacritics for matching without discarding the information a user typed./punct). Deliberately no civic-suffix stripping — Barre City and Barre Town are distinct VT municipalities. This is correctness, not gaming: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. records these as the same place.

   Result (1500 OA rows, real v0.7.2):

   parser	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. BEFORE	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. AFTER	Δ
   neural	94.9%	97.1%	+2.2
   v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.)	93.7%	95.3%	+1.6

   Both parsers rise (the matcher is parser-agnostic), and neural's lead over PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. widens to +1.8pp. 32 neural misses credited — all legitimate (Saint↔St. Johnsbury, Mt↔Mount Pleasant, Butte↔Butte-Silver Bow, Rutland City↔Rutland). Guard verified: all 4 Saint Albans → St. Johnsbury genuine errors still fail (0 wrongly credited) — the matcher trusts disjoint WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. ids, so real ranking bugs stay visible for PR #2.

2. ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. disambiguation — the Saint Albans Town/City → St. Johnsbury cluster (4 rows). A genuine ranking bug: for "Saint AlbansBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure., VT" the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. picks the wrong VT town. Check whether this is name-collision handling or the population/importance tiebreak (must stay ALIGNED with the population/importance ranking used elsewhere — or document why it differs here). This is real resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-depth work, small but clear.

3. Neural parser robustness (feeds the retrain/MLM track, not the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.). Two concrete patterns from (C):

   • <Null> literal tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. in OA input (21695 <Null> 210TH STREET, Holland, IA) → neural fails to parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.; v0 survives. Input sanitization and/or a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. augmentation.
   • DC directional quadrant (6th Street Ne, Washington → neural tags Ne Smith as localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.) and multi-word localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. (Belle Fourche→Belle, Fort Pierre →Fort, SD). CorpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-augmentation candidates for the next train.

4. WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. (lowest priority, after (1)). A handful of tiny localities resolve to nothing (Pennco, Dakota Dunes, Essex Junction Village SD/VT). Extend the custom admin-global-priority.db build rather than chasing these individually.

Geography of failures

VT 39, IA 26, SD 14, MT 11, IL 7, DC 3, CA 1 — concentrated in rural/township-heavy statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. with many small villages and Saint-/Mount-/township name variants. This is consistent with (A) dominating: the failures cluster exactly where canonical-name forms diverge most, not where the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. is structurally weak.

Where neural already wins (bucket: v0-only failures, 25 rows)

v0/PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. returns no localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. for many that neural gets right: Des Moines, Des Plaines (v0 mishandles the Des tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.), Chicago, Georgia, VT (v0 confuses the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. with the stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.), Moretown. These are the cases the neural parser exists to win — and does.