Skip to main content

EU+AU competitive standing (#219)

The 06-23 benchmark put mailwoman ~20pp behind Nominatim on EU @25km, and a cross-harness estimate later put us at ~−4pp. Both compared across different harnesses. This is the same-harness run the night plan asked for: identical held-out OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. rows (real government lat/lon), 40 rows/localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for., every system graded through the same scorer. mailwoman runs v4.15.0 on the -20j candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (the demo's byte-range backend, with the PT/PL/CZ/AU/AT postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. the #370 gate needs).

Resolve-rate @ 25 km (right-locality-area — the headline)​

localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.mailwomanmailwoman+rescorenominatim
IT100%100%73%
PT83%83%45%
PL88%90%98%
AT70%73%98%
CZ90%95%85%
FR95%95%63%
AU75%75%100%
ALL86%87%80%

Two-axis aggregate​

systemn@1km@5km@25kmcond. p50 (km)no-result
mailwoman28028%69%86%2.14%
mailwoman+rescore28029%70%87%2.13%
nominatim28075%79%80%0.017%

What it says​

mailwoman+rescore beats Nominatim on the EU+AU aggregate, 87 vs 80 @25km. The cross-harness estimate had us trailing ~−4pp; the same-harness truth is +7pp ahead. The earlier number was a harness artifact.

The per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. split is the positioning story, not noise. mailwoman wins broad and large where OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. addresses are sparse — IT (100 vs 73), PT (83 vs 45), FR (95 vs 63), CZ (95 vs 85). Nominatim wins the dense-OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. — PL (98 vs 90), AT (98 vs 73), AU (100 vs 75). A geocoder built on a parser + a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. degrades gracefully on the localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. hasn't filled; a search index over OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. data inherits OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names.'s coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. gaps.

The recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. axis is the differentiator. Nominatim is sharper when it answers (75% @1km vs mailwoman's 28% — it returns rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few metres — the highest tier of the geocode cascade. Sourced from address-point and situs data. points, we return admin centroids at p50 2.1km), but it abstains on 17% of inputs where mailwoman misses only 3-4%. mailwoman answers nearly always, at coarser precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1., and — with the isotonic-calibrated per-field confidence — can tell you when to trust the answer. That is the slice a calibrated parser should own.

The #370 rescore lever​

Isolated on the -20j gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (both arms same backend), the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-rescore lever is +1pp aggregate @25km (86→87), zero regressions: CZ +5, AT +3, PL +2; IT/PT/FR/AU flat; no-result 4→3%. The end-to-end span-rescore-e2e run agrees: +1.1pp over n=972, CZ +6. The -20j data did the heavy lifting (the 59→86 climb); rescore adds the last point. It passes the night-plan gate (newly-reachable localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. improve, IT non-regressed, no losers) — a clean default-on candidate, pending the call to flip it.

Artifacts​

  • scripts/eval/competitive-benchmark.ts --candidate-db candidate-global-20j.db --span-rescore → /tmp/bench-full-20j.md
  • scripts/eval/span-rescore-e2e.ts --candidate-db candidate-global-20j.db (n=972)
  • -20j gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. live at R2 mailwoman/gazetteer/2026-06-24a/candidate.db; priors recoverable.