Skip to main content

Competitive benchmark — mailwoman vs Nominatim vs Pelias (2026-06-23)

Same-harness, three-system comparison on real held-out OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. coordinates, run after the #193 GeoNamesGeoNamesA free global gazetteer combining administrative, postal, and POI data across 200+ countries. Supplements Who's On First for postcode centroids and places where WOF has gaps. postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. landed PL/CZ coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. in the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (candidate-global-20i.db). This supersedes the night's two-system (mailwoman vs Nominatim) e2e numbers, which used a different grader (span-rescore-e2e.ts, tag-rank bestCoord) on -20h and so weren't directly comparable to the incumbents.

How we scored​

  • Identical inputs. Every system gets the same raw OA address string and the same countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. hint (mailwoman defaultCountry, Nominatim countrycodes, PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. boundary.country). The PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. arm was countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-scoped for this run — previously it ran unscoped, which understated it by allowing wrong-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. matches.
  • PRIMARY metric — resolve-rate @ 25 km. Within 25 km of truth = right-localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-area; a no-result counts as a miss. Coarse on purpose: mailwoman resolves to admin/postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. centroids, so a km-to-rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. metric would reward the incumbents' rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data.-when-they-match and penalize our centroid. Right-place is the fair test.
  • SECONDARY — conditional median error + the @1/@5 km tierstierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous., to keep the centroid-vs-rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. trade visible rather than hidden.
  • n = 60 rows/localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for., clean OA input (NOT the messy-degradation case). Lever-off (mailwoman) and #370 spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-rescore lever-on (mailwoman+rescore) are both graded from the same parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates..

Resolve-rate @ 25 km​

localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.mailwomanmailwoman+rescorenominatimpeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.
IT100%100%75%85%
PT67%78%48%82%
PL88%90%97%92%
AT73%80%98%100%
CZ85%87%87%78%
FR93%93%65%98%
AU32%35%97%78%
EU (no AU)84%88%78%89%
ALL (incl AU)77%80%81%88%

Two-axis aggregate (incl AU)​

systemn@1km@5km@25kmcond. p50 (km)no-result
mailwoman42024%60%77%2.514%
mailwoman+rescore42026%63%80%2.58%
nominatim42077%80%81%0.017%
peliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.42071%83%88%0.02%

Reading it honestly​

  1. PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. is the strongest system here — 88% @25km all-panel, 2% no-result. A hosted Elasticsearch stack over mixed sources resolves nearly everything and places most of it at rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data.. It is the real bar, and it is ahead of us overall.
  2. On Europe (six EU localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.) mailwoman+rescore beats Nominatim by ~10pp (88 vs 78) and is level with PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. (88 vs 89) — while running as a 30 MB browser modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' with no Elasticsearch. We win IT, PT, FR outright; CZ now edges PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. (87 vs 78).
  3. Australia drags the aggregate. mailwoman 35% vs Nominatim 97 / PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. 78. The whole EU-vs-all-panel gap (88 → 80) is AU. The failure mode is same-named towns scattered across statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., which the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-level postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-consistency gate can't disambiguate. EU-only is the legitimate headline for a Europe comparison because AU is its own, named, unsolved problem — not because hiding it flatters us.
  4. Centroids, not rooftopsrooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data.. mailwoman is 26% @1km vs Nominatim 77 / PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. 71. The @25km parity is right-area parity; the incumbents are far more precise when they hit. StateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. it, don't bury it.
  5. Two levers, partly substitutes. On -20h (no PL/CZ postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.) the #370 rescore lever lifted EU ~+16pp by recovering the fragmented town. On -20i the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. resolves the address directly, so base is already high and the rescore's marginal EU lift shrinks to +4pp. PostcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. (#193) and word-recovery (#370) both close the "silence," by different mechanisms — the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. does most of the work where we now have it; the rescore catches the no-postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. tail (PT, AU).

Levers A + B (post-benchmark, default-off — measured on the same grader)​

The benchmark's failure dump (scripts/eval/failure-dump.ts) classified the 82 misses and pointed at two compounding levers. Both are default-off; numbers are mailwoman+rescore on the same harness/grader, candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. noted.

  • Lever A — postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-disambiguated localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. selection (ResolveOpts.postcodeConsistency, resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.- only, no data/GPU): when a same-named localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. resolves far from a resolved sibling postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one., re-pick the instance nearest the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. (or fall back to the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. point). Fixed the 16 "postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.- available-but-ignored" misses with ZERO regressions.
  • Lever B — extend the #193 GeoNamesGeoNamesA free global gazetteer combining administrative, postal, and POI data across 200+ countries. Supplements Who's On First for postcode centroids and places where WOF has gaps. postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. fill to PT/AU/AT (candidate-global-20j): converts uncovered postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. into anchors, which Lever A then disambiguates. AU 35→65 (+30pp), PT 78→88.
localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.baseline (-20h)+A (-20i)+A+B (-20j)NominatimPeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.
IT1001001007585
PT6778884882
PL8898989792
AT73878798100
CZ8592928778
FR931001006598
AU3235659778
EU (6)6692.594.27889
ALL (7)618490.08188

With both levers, mailwoman leads BOTH incumbents on the @25km right-area metric — EU 94.2 and all-panel 90.0 — from a 30 MB browser modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', no Elasticsearch. The @1km precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. gap (centroids vs rooftopsrooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data.) stands; this is right-AREA, not rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data..

Residuals after A+B (the next levers): AU's remaining misses are mostly a PARSEaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. issue — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emits a house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. as a second postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. ("Grantson StreetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. 51" → postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=51), and the anchor picks the first postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. (51) not the real one (4030); a Lever-A-v2 anchoring on the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. that RESOLVES would catch them. AT's 5 EMPTY are postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. GeoNamesGeoNamesA free global gazetteer combining administrative, postal, and POI data across 200+ countries. Supplements Who's On First for postcode centroids and places where WOF has gaps. AT doesn't carry.

Caveats / next​

  • Clean input only. This run is clean OA strings. The "calibrated parser degrades better than a tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.- matching search index on messy input" claim needs a separate --messy run before it can be made.
  • AU is the open coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present./disambiguation problem (#208 G-NAFG-NAF (Geocoded National Address File). Australia's authoritative open address register (CC-BY-licensed), used as a training source for Australian addresses. ingest + a sub-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. consistency gate).
  • AT (73→80) is the next EU postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. candidate — GeoNamesGeoNamesA free global gazetteer combining administrative, postal, and POI data across 200+ countries. Supplements Who's On First for postcode centroids and places where WOF has gaps. has 18,937 AT rows; the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. has only 809. Same lever as PL/CZ.

Harness: scripts/eval/competitive-benchmark.ts (--span-rescore grades base + lever from one parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.; the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. arm rides the git-excluded diag-geocode-earth.ts, countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-scoped, throttled to respect geocode.earth's 1000/day + 10/s). Raw: candidate-global-20i.db, modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' out/v191/model.onnx (v4.13.0).