A precision lever you can dial — calibrated confidence on messy input (v4.13.0)
2026-06-24. Shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' neural-weights-en-us v4.13.0 (int8), calibrated via the v4.13.0 isotonic table. 472 held-out OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. coordinate goldens across us/it/pt/pl/fr/au (≤80/localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.), each perturbed to be messy: lowercased, comma-stripped, common streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. words abbreviated, dash-postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. dropped — the house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. untouched. Coordinate-graded right-place @25km ("no result" counts as a miss). Per-result confidence = the minimum calibrated confidence across the resolved nodes (a coordinate is only as trustworthy as its least-sure driving component).
The claim
A geocoder that returns a best guess gives you one number and no way to know which answers to trust. mailwoman returns a coordinate and a calibrated confidence, so you can set a threshold τ and accept only the answers it is at least τ confident about. As τ rises, precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. rises and recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. falls — a lever no search index exposes. The question this report answers: does that lever actually work — does higher confidence mean higher right-place rate, and does it hold on data the curve was not drawn on?
The lever (draw split, 236 rows)
Sweep τ; at each, precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. is right-place @25km among the answers at or above τ, recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. is the fraction of all rows answered at or above τ.
| τ | accepted | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. @25km | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. |
|---|---|---|---|
| 0.00 | 159 | 84.3% | 67.4% |
| 0.70 | 147 | 85.0% | 62.3% |
| 0.80 | 115 | 92.2% | 48.7% |
| 0.90 | 79 | 93.7% | 33.5% |
| 0.94 | 37 | 97.3% | 15.7% |
| 0.97 | 29 | 96.6% | 12.3% |
PrecisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. climbs from 84.3% to 97.3% as the threshold rises; recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. is the price, falling from 67% to 16%. The confidence is not decoration — it ranks answers by how likely they are to be right.
The honesty check (held-out 50%, 236 rows the curve was not drawn on)
A confidence story a judge can break is worse than none. Split the rows at the draw-split median confidence (0.900) and re-measure right-place on the held-out half:
| held-out bucket | n | right-place @25km |
|---|---|---|
| confidence < 0.900 (low) | 68 | 72.1% |
| confidence ≥ 0.900 (high) | 92 | 85.9% |
The high-confidence bucket outperforms the low-confidence bucket by 13.8pp out-of-sample. The discrimination is a property of the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', not of the slice the curve was fit on.
Where the confidence comes from (per-locale, τ=0)
| localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. | n | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. @25km | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | reads as |
|---|---|---|---|---|
| US | 80 | 94% | 85% | covered — precise and confident |
| IT | 80 | 94% | 99% | covered |
| FR | 80 | 91% | 100% | covered |
| PT | 80 | 76% | 31% | building — correctly less confident |
| PL | 80 | 28% | 36% | building — correctly low confidence |
| AU | 72 | 63% | 53% | building |
The discrimination is the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' flagging its own coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.. Where mailwoman has gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. depth (US, IT, FR) it is both precise and confident; where coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. is still being built (PL, PT, AU) it is correctly unsure, so a τ threshold removes exactly those answers. That is the asset: for a caller who must avoid wrong answers — a record-matcher deduping compliance data, say — the threshold cuts the error rate by trusting only the answers the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' stands behind. CoveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. breadth is a separate axis, tracked elsewhere.
What this report does NOT claim
The plan opened as a headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. against Nominatim on the same messy set. That comparison is withheld: the Nominatim fetch hit rate-limiting during crash-restarted runs (AU returned nothing for all 63 valid AU addresses, FR 45% null, PT 38% null), so the competitor's recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. and precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. are unreliable here and any "mailwoman wins" read would be a rate-limit artifact. The clean competitive result stands from the 2026-06-23 benchmark (US right-place @25km, mailwoman 99% vs Nominatim 84%, #775) as supporting context, not as tonight's measurement. A spaced, policy-respecting re-fetch is the next step if the headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. on mess is wanted.
Reproduce
node scripts/eval/confidence-discrimination.ts \
--locales us,it,pt,pl,fr,au --n 80 --rows-out /tmp/rows.jsonl --out scorecard.md --svg curve.svg
# re-analyze instantly from the cached rows (no re-parse, no API):
node scripts/eval/confidence-discrimination.ts --rows-in /tmp/rows.jsonl --agg min
The harness separates collection (parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. + resolve + grade, the expensive part) from analysis (sweep + plot), checkpointscheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. every row so a crash resumes, and rate-limits Nominatim only on a cache miss.