Reading our numbers

The question

A geocoder reports that it resolved 98% of your file. What do you now know? Less than the sentence suggests, and the gap is where geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. benchmarks go wrong — including ours, four times, on the two pages next door.

This page is the reading guide for those pages. It is also the checklist we hold our own panels to, which is why each trap below is illustrated with one of our own numbers rather than someone else's.

Trap one: the resolve rate is not the accuracy

A resolve rate counts rows that came back with a coordinate. It says nothing about whether the coordinate is right, and nothing about how right.

Every geocoder has a fallback, and ours is no exception. On 2026-08-04 we sent five invented streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. names in real French cities through the same configuration the French panel uses — starting with 12 Rue Inexistante, 75002 Paris. All five returned a coordinate. The Paris row came back at 48.857, 2.343, which is the Paris commune centroid — 1.281 km from the centroid of the 3,386 BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. address pointssitus dataA dataset of exact address-point coordinates (rooftop-level). Mailwoman's geocoder uses a national situs layer (124.9M US points built from state address-point sources) as the highest-precision tier of the geocode cascade. carrying the 75002 postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one., and an unbounded distance from a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. that does not exist.

A null would have been less useful to most callers, so the fallback is the right behavior. It also means a file of invented streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. names in real cities scores 5 out of 5 on a resolve rate.

So the resolve rate and the precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. are two numbers, and the one that carries the meaning is the tier: which artifact answered.

Tier	What answered	What the number means
address_point	A rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. or parcelparcelA property polygon or record carrying a situs (site) address and often a separate owner mailing address. County GIS parcel aggregations are a training source for address-point variety and situs-vs-owner divergence. registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise.	The building
interpolated	An estimate along the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.	The block
street	A streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. centroid	The streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.
admin	A commune or locality centroidlocality centroidThe representative center point of a city or locality, used as a coarse coordinate when no exact address point is available — the coarsest tier of the geocode cascade.	The town

Our Belgian panel reports 30 of 30 in its best arm. Every one of those 30 answered at admin. The panel's own summary says so on the line under the score, because a reader who takes 30 out of 30 for streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. accuracy has been misled by a true number.

There is a second edge on this, and our own script fell off it. A tier field reports where the resolution cascade ended, not whether it produced anything — so a row that returned no coordinate still carries admin. Counting the field directly put three unresolved rows in the admin column of each of the first two Belgian arms, and the published table read admin: 30 two lines under returned a coordinate: 27. The two rows disagreed about the same three addresses. The fix is to bucket on the tier that answered and count the rest as none, which is what the table shows now.

What to ask of any benchmark, ours included: what fraction answered at each tier, and do the tier counts add up to the resolve count? A published resolve rate with no tier breakdown is not measuring what the reader thinks, and a tier breakdown that outnumbers the resolve rate is counting silence.

Trap two: circularity

If the evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set and the reference data come from the same place, a high score is partly a statement about bookkeeping.

The French panel is the clean example. The French rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data. tier is the Base Adresse Nationale; the panel draws its addresses from BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. and grades against BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.'s coordinates. Ninety- four of a hundred rows land within one meter, which sounds like a precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. result and is not one. It means the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. found the row it was looking for, in the table it was looking in. What that measures is the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and the routing. What it cannot measure is whether BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.'s coordinate for the building is correct, because the panel has no second opinion about the building.

This is not a reason to discard the number — parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and routing are most of what goes wrong, and a panel that isolates them is worth running. It is a reason to state the claim at the size it earned: "a French address that exists in BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. resolves to its BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. rooftoprooftopGeocoding precision at the building or parcel level — coordinates within a few meters — the highest tier of the geocode cascade. Sourced from address-point and situs data.", not "our French geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. is accurate to one meter".

What to ask: where did the evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. addresses come from, and where did the reference coordinates come from? If the answer is the same dataset, the number is about retrieval and not about accuracy.

Trap three: a panel is not a population

The French page reports on 100 addresses. The Belgian page reports on 30. Neither is a sample from which a national figure can be computed, and neither page computes one.

Two things go wrong when a panel is read as a population.

The confidence interval is wider than the digits suggest. At n = 30, a score of 24 has an uncertainty of several rows in either direction. Reporting it as 80.0% adds a decimal place the panel did not buy. We report counts rather than percentages on the panel pages for that reason.

The panel's composition decides the answer. Our Belgian rows are civic addresses on well-known streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. — an easier population than a delivery manifest with apartment numbers and typos. Our French rows are one per postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one., which spreads them across 58 departments and thins out the dense urban cases a uniform draw would have produced. Both choices are defensible. Both change the number, and neither page claims to know by how much.

A panel earns statements about the addresses in it and about the defects those addresses exposed. Both of our panels found real defects that way, and that is the work a panel is good for.

What to ask: how large is n, how were the rows chosen, and does the reported precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. exceed what n supports?

Trap four: throughput does not survive the host

Our panel pages report elapsed time — 2,615 ms for 200 French geocodes, 4,273 ms for 90 Belgian ones. Those numbers describe one machine doing one thing.

Three mechanisms break the extrapolation:

• Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' load is excluded, and it is not free. Both figures start after the weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. are in memory. A process that loads the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' per request measures something else entirely.
• InferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. blocks the JavaScript threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases.. onnxruntime-node's session.run() runs synchronously on the event loop rather than on a worker pool, so a second concurrent request does not get its own free core, and a smaller host does not scale down linearly. Geocode a large file at volume has the measurements.
• The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. is read from disk. A warm page cache and a cold one give different numbers for the same query, and our panels run warm — the same shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. were open moments earlier.

Timing here is a sanity check on the order of magnitude, not a throughput specification. Measure on your own hardware, with your own data root, at your own concurrency.

What to ask: was the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' load counted, what was the concurrency, and was the cache warm?

What we hold ourselves to

Every number on a benchmark page here comes from a run of a script published beside it, on inputs published beside it, with the result file published beside it. That is the standard the pages are written against, and it is checkable rather than promised: if a page reports a count, the file it came from is one click away and carries the per-row records behind it.

Three things follow from that, and all three are visible on the pages:

Lossesloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. get the same room as wins. The French page publishes six failing rows with their distances and their broken parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.. The Belgian page publishes six, plus a row that scores as correct while carrying a wrong parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. underneath. A page that reports only the aggregate is not showing you the thing you would have had to debug.

A result file names the versions it ran against. Each one records versions.mailwoman, versions.model — the resolved artifact filename, not the package alias — and versions.gazetteer. A reader whose re-run disagrees with ours needs those three to tell data drift from code drift, and without them a differing number is uninterpretable rather than informative. The French file also records its BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. release, on the committed panel, because that is a property of the addresses.

Instrument defects get published too. Four so far, all in the Belgian script, all found after its first numbers were written:

1. It collapsed the parsed and resolved localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. checks into one metric, scoring a row that landed in Slovenia as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. hit because the parsed spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. still read correctly.
2. It dropped an unmeasurable bilingual pair from its own denominator, turning a row that returned no coordinate into a clean 4-of-4.
3. It counted a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. name match as a resolution without checking the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., scoring a Belgian commune name matched in the Netherlands and another matched in France as hits — on a panel whose subject is cross-border misrouting. That one cost two rows in each of two arms.
4. It read the tier field alone, so a row that returned no coordinate still carried admin — three rows in each of the first two arms, which is why an early table read admin: 30 one line under returned a coordinate: 27.

Each is fixed in the published script, and each is named on the page rather than corrected without a note, because a benchmark's instrument is part of its result. The pattern across all four is the same: the metric counted something adjacent to the claim and the aggregate looked reasonable, so only reading the per-row records caught it.

Related

• France — 100 addresses from the BAN — the circularity case.
• Belgium — 30 addresses, three configurations — the tier case.
• How close is close enough? — deciding the tolerance a distance should be read against.
• Calibration and confidence — what the per-result confidence number is, and is not.