EU qualified-name recall — measuring #734's collision-risky lever

Two threadsthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. this shift kept pointing at the same thing. The spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-rescore investigation (#370, PR #777) found its swap tail full of Portuguese localities the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emitted as Santa Eulália Viz and Nogueira Do Cravo Ohp. And #734, profiling the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.'s EU recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1., had already named the pattern: OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. writes the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. in its qualified form — Hart b.Graz, Roche VD, Lenk im Simmental, Santa Eulália Viz — while the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. holds the base name. #734 proposed a base-name normalization to close the gap and flagged it: "collision-risky, measure first."

This measures it.

The setup

For each EU coord-golden row (the same 7-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. panel #370 ran on), take the gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. and exact-match it against the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-constrained. For a miss, strip the qualifier and re-match — but grade the recovery by coordinate, not string (the #566 discipline): the recovered place must land ≤25 km from the row's truth point. A near hit is a real recovery; a far hit is a same-name collision where the strip ate too much and matched a different place. Grading the string alone would score both as wins, which is exactly the failure mode "measure first" is guarding against.

Two normalization tierstierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous., measured separately because their risk is not the same:

• Structural — strip /X, b.X (German bei), im/an der/ob der X. Unambiguous Austrian/Swiss disambiguation suffixes; the base tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. survives intact.
• Trailing-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. — additionally strip a trailing short capitalized tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. (VD, S, Viz, Ohp), bounded to ≤3 characters. This is the one #734 called risky: it recovers Roche VD but could eat a real name part (Santa Cruz → Santa).

The result

baseline exact recall          = 883/972 (90.8%)
+ structural (/  b.  im/an der) = +0 near, 1 collision   → 90.8%  (low-risk)
+ trailing-token (VD/S/Viz/Ohp) = +14 near, 0 collisions → 92.3%  ("risky")

Two things flip the intuition.

First, the headline recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. is 90.8%, not the recorded 88% — confirming #734's read that the old number was depressed by a Lithuanian evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-extraction artifact, not a real coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. hole. (Lithuania isn't in this panel, so it can't drag the number here.)

Second, the risky lever is the safe one, and the safe lever does nothing. The structural suffixes (b., im) recover zero rows here — those are Austrian/Swiss forms that don't appear in this panel (an AT/CH panel would be needed to measure them). The trailing-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. strip, the one flagged collision-risky, recovers 14 Portuguese localities at zero collisions:

"Ferreiros Amr"        → "Ferreiros"        (0 km)
"Santa Eulália Viz"    → "Santa Eulália"    (1 km)
"Nogueira Do Cravo Ohp" → "Nogueira Do Cravo" (1 km)   ← strips "Ohp", keeps "Do Cravo"
"Vilar Chão Afe"       → "Vilar Chão"       (0 km)
"Pereiro Act"          → "Pereiro"          (0 km)
"Paradela Mdr"         → "Paradela"         (0 km)

These are Portuguese freguesia/concelho abbreviations — a 3-letter administrative code OA appends for disambiguation (Viz = Viseu, Amr = Amares, Ohp…). The base name resolves on top of the truth point. And the collision the "risky" framing feared — eating Cruz off Santa Cruz — doesn't happen, because the ≤3-character bound separates the codes (Viz, Ohp) from real name parts (Cravo, Chão, Cruz, all ≥4). The bound is the whole game; without it the lever would be risky, with it the measured collision rate is 0/14.

What this says

• The candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.'s EU localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. is already ~91% on this panel (matching #734's "real ~93%"); the qualified-name gap is real but small.
• A trailing-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. base-name normalization bounded to ≤3-char codes is a clean, collision-free lever worth +1.4 pp here — almost entirely Portuguese freguesia codes. It is the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-side twin of what #370's spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-rescore (PR #777) does parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.-side: both recover the same Santa Eulália Viz rows.
• The structural suffixes (b., im, an der) are unmeasured here for lack of an AT/CH panel; #734's examples suggest they exist, so a follow-up should grade them on Austrian/Swiss holdouts before building.
• Don't widen this past ≤3 chars without re-measuring — the safety is entirely in the bound, and a 4-char allowance would start eating Cruz.

The harness (scripts/eval/eu-qualified-name-recall.ts) reports per-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1., both tierstierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous., and the coordinate-graded collision count; DEBUG=1 dumps each recovery. Reusable for the AT/CH follow-up and any future candidate rebuild.