Competitive benchmark — mailwoman vs Nominatim vs Pelias (2026-06-23)
The honest headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads., run for trade-show readiness. Harness: scripts/eval/competitive-benchmark.ts.
Reproduce: node scripts/eval/competitive-benchmark.ts --n 150 --locales it,pt,pl,at,cz,fr,au (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. via geocode.earth; Nominatim via the public API at ~1 req/s).
Method (and why it's built this way)
Three systemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed., identical real held-out OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. rows (truth lat/lon), 150/localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. The systemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. return structurally different things — mailwoman resolves to a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. centroid; Nominatim returns the matched OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. object (rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. when it matches, nothing when it doesn't); PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. is ES over OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names.+OA+WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. — so a raw median-error race flatters whoever returns rooftopsrooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. on the addresses they match and hides who returns nothing. So we score resolve-rate @ a coarse km threshold (within Xkm of truth; "no result" = a miss) as the honest denominator, plus conditional accuracy. Coarse thresholds (5/25 km = "right localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. area") because mailwoman returns centroids — a km-to-rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. metric would unfairly reward rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data.-when-it-matches.
Headline — resolve-rate @ 25 km (clean inputs)
| localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. | mailwoman | nominatim | peliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. |
|---|---|---|---|
| IT | 92% | 75% | 79% |
| PT | 57% | 47% | 65% |
| PL | 42% | 96% | 92% |
| AT | 73% | 97% | 89% |
| CZ | 33% | 88% | 68% |
| FR | 66% | 59% | 94% |
| AU | 38% | 97% | 76% |
| ALL | 59% | 79% | 81% |
| system | n | @1km | @5km | @25km | cond. p50 | no-result |
|---|---|---|---|---|---|---|
| mailwoman | 972 | 26% | 52% | 59% | 1.8 km | 27% |
| nominatim | 972 | 74% | 78% | 79% | 0.0 km | 20% |
| peliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. | 972 | 66% | 77% | 81% | 0.0 km | 1% |
On clean OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries., mailwoman trails both (59 vs 79/81). The losing rows stay in the table.
What's real and what's confounded (verify-before-verdict)
- The lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. is NOT a resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-config handicap. mailwoman is ~44% on the trailing localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. across all three resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. configs — admin-only, admin +
postcode-locality-intl, and the demo's actual candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (candidate-global-20h). The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. isn't the cause. - Our internal "resolve-rate" metric OVERSTATESregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. by ~15–22 pp. Internal panel: PL resolve 62%, CZ 52%, AU 53%. Honest @25 km right-place: PL 42%, CZ 28–33%, AU 32–38%. The gap = resolves that land >25 km (regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-level fallback / wrong same-name place) — counted as success by resolve-rate, as a miss by right-place. We have been grading ourselves on a lenient metric. (mailwoman's centroid is NOT the issue — p50 1.8 km, well inside 25 km; @25 km forgives it.)
- The test set favors PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.. OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. is one of PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s INDEXED sources — its 81% / p50 0.0 km is partly recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.-of-its-own-data, not generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure. (home-field advantage). Nominatim (OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names.) overlaps less. mailwoman trained on a disjoint held-out split.
- The real mailwoman gap is coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present./recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.: ~27% no-result aggregate (worse on EU non-IT) vs PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. ~1% / Nominatim ~20%. It fails to return a usable coordinate for a quarter of these addresses.
Messy inputs — the slice that should favor a calibrated parser
(filled from the --messy run: lowercase + dropped commas/dash-postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. + abbreviations — the "typed in a hurry" case where a search index that leans on exact tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. + structure should degrade more than a learned parser.)
Messy = lowercase + dropped commas/dash-postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. + abbreviations (house numbershouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. preserved). 40/localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for..
| system | clean @25km | messy @25km | Δ |
|---|---|---|---|
| mailwoman | 59% | 49% | −10 (graceful) |
| nominatim | 79% | 81% | +2 (robust free-text) |
| peliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. | 81% | INVALID — rate-limited |
⚠ PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s messy "6%" is a geocode.earth RATE-LIMIT artifact, not a finding — verified by direct query: the API now returns HTTP 429 on every call, clean OR messy (the trial key's quota exhausted after ~1.2k calls; the messy run ran later than the clean run, so its PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. column collapsed to 429→null→"no result"). The clean PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. 81% (earlier, under quota) is likely valid; the messy PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. number is discarded. verify-before-verdict caught what would have been a false "PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. collapses on messy" headline.
Real messy takeaway: mailwoman degrades gracefully (−10pp); Nominatim's free-text search is robust (doesn't degrade). My perturbation didn't break Nominatim — OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. has the addresses and its tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. forgives lowercase/no-comma/no-postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one..
US — the home turf (mailwoman vs Nominatim; Pelias rate-locked)
| localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. | mailwoman | nominatim |
|---|---|---|
| US | 99% | 84% |
| system | n | @1km | @5km | @25km | cond. p50 | no-result |
|---|---|---|---|---|---|---|
| mailwoman | 150 | 18% | 67% | 99% | 3.2 km | 0% |
| nominatim | 150 | 82% | 83% | 84% | 0.0 km | 16% |
On US, mailwoman dominates: 99% vs 84%, and 0% no-result vs Nominatim's 16%. Nominatim returns rooftopsrooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. when it matches (p50 0.0 km) but misses 16% of US addresses — OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names.'s US coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. gaps (rural, new developments). mailwoman (TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data. + national situssitusThe physical site address of a property, as opposed to the owner's mailing address. Parcel records often carry both; the divergence is a real-world data-quality challenge. + the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.) resolves every address to the right localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.. The US set is OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries., which Nominatim (OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names.) does NOT index wholesale, so this is genuine coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. superiority, not data overlap.
Honest verdict
The picture is nuanced — and good, once you stop grading on the lenient metric:
| market | mailwoman | Nominatim | PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. |
|---|---|---|---|
| US | 99% | 84% | (rate-locked) |
| EU (clean) | 59% | 79% | 81%¹ |
- ¹ EU is OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries., a PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-indexed source — its 81% is partly recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.-of-its-own-data.
Where we win: US accuracy + coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. (the high-value market) — 99 vs 84, zero misses. Plus two capabilities the competitors structurally lack: calibrated confidence (knows when it's wrong) and deployability (30 MB, in-browser/offline, no Elasticsearch, no PostgreSQL+OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names.-planet).
Where we trail: EU coordinate coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.. Nominatim/OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. and PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor./OA simply have more EU address data than our gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. + modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' do today. This is a coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present./recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. gap (no-result), not a precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. gap (our resolved p50 is 1.8 km) — and it's the active roadmap: the v4.13.0 multi-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. ship, #370 (the rescore that recovers the wrong-place tail), and G-NAFG-NAF (Geocoded National Address File). Australia's authoritative open address register (CC-BY-licensed), used as a training source for Australian addresses./coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. ingestion all target exactly it.
The methodology correction (most important internal takeaway): our internal "resolve-rate" metric overstated EU by ~15–22 pp — it counts regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-level / wrong-same-name resolves that land >25 km from truth. Going forward, grade on right-place resolve-rate (@25 km / PIP-containment), not bare resolve-rate. This is the #566 "grade the assembled coordinatecoord metricThe primary evaluation metric: distance from the resolved coordinate to the true address point. Measured at percentiles (p50, p90) and as 'within X meters.' Prevents the label-F1 trap where a model scores higher on token labels but geocodes worse." discipline, sharpened.
Trade-show framing: lead with US dominance + the calibrated-confidence demo + deployability; present EU as the fast-improving frontier (just shipped 16 localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.). Do not claim "more accurate than Nominatim" globally — it's false on EU and true on US; claim it precisely.
Caveats: PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. rate-locked tonight (US PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. + messy PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. untested — re-run paced on fresh quota); EU set favors PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. (OA overlap); n=150/localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for..