#370 — recovering the EU no-result tail with a span-rescore
The competitive benchmark (the vs-Nominatim/PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. scorecard, PR #775) put a number on where we lose to Nominatim and PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. in Europe, and it wasn't precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.. When mailwoman resolves an EU address its centroid is tight (p50 1–3 km). The gap is no-result — on the messy multi-order EU panel, mailwoman returns nothing for ~45% of addresses where the competitors return something. So the question for closing the gap isn't "place it more accurately." It's "place it at all."
This is the story of one lever against that tail: a post-hoc spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-rescore (#370). Falsify first, then build — and let the coordinate, not the labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., decide.
Where the tail comes from
Run the shipped v4.13.0 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' over a 7-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. coordinate-golden panel (IT/PT/PL/AT/CZ/FR/AU, real OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. points) and 259 of 972 rows come back unresolved. Bucket each one by why:
| bucket | meaning | count |
|---|---|---|
| swap | gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. IS in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., but the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emitted a different/fragmented tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. | 116 |
| needsK | gold in gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emitted no localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. at all | 34 |
| coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.-gap | gold NOT in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. — nothing to recover | 104 |
| emitted-but-unresolved | modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emitted the gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. still missed | 5 |
The 116 swaps are the prize. A representative one: 86-300 Grudziądz, Daliowa 4. The gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. is Grudziądz, which is in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. — but the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' splits it into Grudzi + dz on the ą combining mark (the #555 fragmentation), so neither fragment resolves. The full word sits right there in the raw text, intact. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s subword tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. mangled it; a whitespace tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. wouldn't.
Falsify before building
Before writing a line of the rescore, the question that could kill it: when the gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. does resolve, does it land anywhere near the truth — or is "Grudziądz" one of five Grudziądzes and we'd be swapping in a coin flip? That's the #566 trap in a new costume: a labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. that matches gold but a coordinate that's 300 km off — grade the assembled coordinatecoord metricThe primary evaluation metric: distance from the resolved coordinate to the true address point. Measured at percentiles (p50, p90) and as 'within X meters.' Prevents the label-F1 trap where a model scores higher on token labels but geocodes worse., never the labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag..
So for all 116 swaps, resolve the gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. with the row's postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. as the disambiguator and measure great-circle to truth:
top-1 (resolver-ranked, postcode-disambiguated): p50 1.8 km · p90 12 km
best-of-5 (ceiling if disambiguation is perfect): p50 1.4 km · p90 5 km
p50 1.8 km. The swap is real coordinate-recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1., not a same-name mirage. Build it. (And note the division of labor the buckets already imply: PL/CZ/AT own the swap tail; IT and FR have zero swaps — their unresolved rows are pure coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. gap, a job for more gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., not a rescore.)
The build, and the knob that was backwards
The rescore enumerates contiguous raw-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. (diacriticsdiacriticAn accent mark that modifies a letter (é, ñ, ç). Address normalization must fold diacritics for matching without discarding the information a user typed. intact, unlike the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s subwords), skips any spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. overlapping a confident streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. / house-number / postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. constituent, and exact-matches each against the same-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.. The open question the falsifier couldn't answer: with no gold to point at, can enumeration find the right spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. on its own?
First cut, shortest-spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-wins (the textbook over-merge guard — prefer "Santa Maria" over "Santa Maria da Feira"): 68% of the tail recovered, but only 49% of recoveries were the right localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., and p90 blew out to 359 km. Dumping the wrong ones showed the heuristic was backwards for this data:
gold "Tomaszów Mazowiecki" → recovered "Tomaszów" (135 km away — a different Tomaszów)
gold "Nogueira Do Cravo Ohp" → recovered "Nogueira" (139 km)
gold "Póvoa De Rio De Moinhos" → recovered "Póvoa" (207 km)
On real OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. the gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. is the longer, more specific name, and shortest-wins kept grabbing its ambiguous prefix. Flip it to longest exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness. wins and the specific name beats its own prefix:
| metric | shortest-wins | longest-wins |
|---|---|---|
| right localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. (gold string) | 49% | 71% |
| coord p50 | 5.7 km | 3.0 km |
| coord p90 | 359 km | 206 km |
PL — the biggest swap localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. — goes to 56 of 56 recovered, all gold, p50 1.8 km, p90 8 km. Solved.
Grade the coordinate, not the string
71% "right" by gold-string undersells it, because the string metric counts Santa Eulália against a gold of Santa Eulália Viz as a miss — when it's a 1 km miss. The honest read is the coordinate, at the benchmark's right-place bar:
coordinate quality of the 175 recoveries:
≤10 km 128 (73%)
≤25 km 136 (78%)
≤100 km 142 (81%)
>100 km 33 (19%)
The rescore lifts 136 of 259 unresolved EU rows — 53% of the entire no-result tail — to a right-place coordinate (@25 km). That's a direct dent in the one number the benchmark said we lose on. The cost is 33 mis-fires past 100 km, and they aren't spread evenly: they're the coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.-gap localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. (IT, AU) where the gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. isn't in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. at all, so the rescore latches onto some other tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. that happens to name a far-off town. AU adds its own wrinkle — same-name towns across statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. (Springfield QLD/NSW/VIC) that need stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. agreement, not just countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head..
The consistency gate
The mis-fires share a cause: nothing checks that the recovered localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. is geographically consistent with the rest of the address. So the validator carries a postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. gate — resolve the row's postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. to a point and reject any candidate match that lands more than 50 km from it. The postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.→point lookup the admin gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. can't do (it returns nothing for an EU postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.) the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. can: it folds postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. in as queryable rows. The gate is conditional by construction — when the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. doesn't resolve, it can't fire and the rescore falls through unchanged, so it never hurts a localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. it can't help.
With the gate on, the IT story flips cleanly: from 4 recoveries all wrong (p50 207 km) to 2 recoveries both near (p50 22 km) — it killed the far false-positives and kept the near matches. Aggregate p90 drops 206 → 145 km; the >100 km mis-fires fall 33 → 29; the right-place count holds (136 → 137).
| metric (175→172 recovered) | gate off | gate on (50 km) |
|---|---|---|
| right-place @25 km | 136 | 137 |
| coord p50 | 3.0 km | 2.9 km |
| coord p90 | 206 km | 145 km |
| >100 km mis-fires | 33 | 29 |
The gate's reach is bounded by where the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. has postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.: IT resolves 97% of them, so its false-positives get caught; CZ and AU resolve ~none, so their mis-fires survive (AU's are the cross-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. same-name problem — Springfield in three statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. — that needs stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. agreement the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. gate can't supply). That's a coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. limit, not a design flaw: the gate's reach grows automatically as the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. fills in, with no code change.
Production wiring + the end-to-end result (#780)
The wiring is now done (#780): spanRescore lives in resolveTree behind a default-off ResolveOpts.spanRescore — the idiomatic twin of the addressPoints/interpolation tierstierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous., injecting the recovered localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. via the same decorateNode path, gated on hasResolvedPlace (the #685 brake). And the wired path is graded end-to-end (scripts/eval/span-rescore-e2e.ts): parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. → resolveTree with the flag off vs on, same modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' + candidate backend, the flag the only variable.
EU coord panel (n=972, @25km right-place), spanRescore off → on:
resolved: 76.0% → 95.2% (+19.2pp)
right-place @25km: 63.2% → 79.2% (+16.0pp)
PL 43→85 · CZ 29→71 · PT 62→73 · AT 79→85 · AU 35→42 · IT/FR flat
~17% of new resolutions land >25km — the mis-fire rate, trimmed by the gate where postcodes resolve.
The strategic read, stated precisely (a trade-show claim has to survive a hostile check). The clean, defensible result is the +16 pp lift — a flag-only A/B that substantially narrows the EU no-result tail. What it does to the standing-vs-Nominatim is more nuanced than "parity," and the per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. picture is the honest version. Against #775's Nominatim on the same panels:
| IT | PT | FR | PL | AT | CZ | AU | |
|---|---|---|---|---|---|---|---|
| mailwoman + lever | 99 | 73 | 81 | 85 | 85 | 71 | 42 |
| Nominatim (#775) | 75 | 47 | 59 | 96 | 97 | 88 | 97 |
So with the lever on, mailwoman leads on IT/PT/FR and still trails on PL/AT/CZ (narrowed, not closed) and AU (the cross-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. problem the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. gate can't touch). The aggregates land ~79% each — but that's a mix (the IT/PT/FR leads offset the Slavic/German/AU trails), not a uniform catch-up. And it's a cross-harness comparison: the e2e grades mailwoman more leniently than #775 did (e2e IT 99 vs #775's 92), so a same-harness run could put mailwoman-with-lever below 79%. The honest claim: the lever makes mailwoman competitive on EU aggregate and ahead on several countries — not "matches Nominatim across EU." Confirm the aggregate standing with a single-harness mailwoman-vs-Nominatim run before claiming parity.
One process note: the first e2e run reported +0.0 pp because @mailwoman/core resolves to the compiled out/ (stale, pre-#780) while the unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. tests alias to source — yarn compile, then the real +16. Grade the wired path on compiled code, not only source unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. tests.
What stays off
The rescore stays default-off. What remains before default-on is a decision, not a build: flip ResolveOpts.spanRescore (the +16 pp recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. against the ~17% >25 km mis-fire is the trade), and widen postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. so the gate reaches CZ/AU (IT today). The harness — rescore-ceiling-probe.ts (falsifier), span-rescore-validate.ts (standalone), span-rescore-e2e.ts (wired) — regrades any of those changes.