Benchmarks

Every number on these pages came out of a script published beside it, run on inputs published beside it, writing a result file published beside it. Our numbers ship with the code to re-run them, and the code is here rather than described.

Two panels are published so far. Each is a panel, not a national accuracy figure, and each says so.

I want to see what a national register buys

France has one — the Base Adresse Nationale — and the French rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. tier is built from it. A hundred BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. addresses, rendered in two surface forms, graded against BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.'s own coordinates: 96 of 100 land within a kilometer in the canonical order, 97 of 100 with the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. moved to the front. Four rows lose part of the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. name in the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and two land on a same-named streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. in a neighboring commune.

The page states the circularity in the same breath as the score, because grading a BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.-backed tier against BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. measures parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and routing rather than coordinate accuracy.

France — 100 addresses from the BAN

I want to see what happens without one

Belgium has no rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise., no localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. of its own, and no evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. tier. Thirty bilingual Belgian address lines through the global candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture., in three configurations: 24 of 30 land in Belgium out of the box, and 30 of 30 once the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. is pinned. Six rows go wrong without the pin, five of them for one reason — a countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. prior that filters the right answer out before ranking runs.

Belgium — 30 addresses, three configurations

I want to know how to read a geocoding benchmark

Four ways one reads better than it is: a resolve rate that counts fallbacks as successes, an evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set drawn from the reference data it is graded against, a panel quoted as a population, and a timing figure that does not survive a change of host. Each is illustrated with one of our own numbers.

Reading our numbers

Running them yourself

run-all.sh pulls the two data bundlesevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. the panels need — the 1.65 GB global gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. and the 6.95 GB French rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. — and runs both panels in one pass:

npm install mailwoman @mailwoman/neural @mailwoman/neural-weights-en-us \
            @mailwoman/neural-weights-fr-fr @mailwoman/resolver \
            @mailwoman/resolver-wof-sqlite @mailwoman/ban @mailwoman/core @mailwoman/spatial
curl -fsSLO https://mailwoman.sister.software/benchmarks/run-all.sh
curl -fsSLO https://mailwoman.sister.software/benchmarks/fr-ban-panel.mjs
curl -fsSLO https://mailwoman.sister.software/benchmarks/fr-ban-sample.json
curl -fsSLO https://mailwoman.sister.software/benchmarks/be-panel.mjs
curl -fsSLO https://mailwoman.sister.software/benchmarks/be-panel.json
bash run-all.sh ./benchmark-data

The Belgian panel needs only the first bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line.; run it alone with node be-panel.mjs if 6.95 GB of French rooftop datasitus dataA dataset of exact address-point coordinates (rooftop-level). Mailwoman's geocoder uses a national situs layer (124.9M US points built from state address-point sources) as the highest-precision tier of the geocode cascade. is not what you came for.

Your numbers will differ from ours where the versions differ — a newer BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. release moves rows, a newer gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. moves rankings, and a newer modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' moves parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.. Each result file records a versions block naming the Mailwoman version, the resolved modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' artifact and the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. file it ran against, so a disagreement can be attributed rather than only observed. The French panel additionally records its BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. release, on the committed sample file; the Belgian panel records no data release, because there is no Belgian registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. for it to record one from.

Ours ran against Mailwoman 8.7.0, model.onnx from neural-weights-en-us@7.0.0, and candidate-global-1026.db. Note that npm currently serves 8.6.0, so an unpinned install is a version behind these pages.

Earlier versions of these pages named that modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' file model-v401-base-step-060000-int8.onnx. Same bytes — md5 c968c24ad1a3e59658ff1d92c5720575 either way — but that name only ever existed on the machine that trained it, reaching the result files through a development symlink. model.onnx is what the weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. package publishes and therefore what your own re-run will stamp.

What is not here

No comparison against another geocoder's output. Benchmarking someone else's service means running it under terms we did not write, on a query mix we chose, and publishing an accuracy number for a system we do not maintain. Compare sets out what each of the alternatives is built for and when to choose it, without claiming an accuracy figure on its behalf.