Skip to main content

Calibration and confidence

The question​

Every spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. in a parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. carries a confidence. A localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. at 0.92, a regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. at 0.88. The number looks like a probability — it sits between 0 and 1, and the numbers across the competing labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. at one position sum to 1 — so the natural reading is "right about 92% of the time".

That reading is wrong, and the direction it is wrong in is not the one people expect. This page is about what the number reports, why its scale is off, and what a correction table does to fix it.

The analog​

Every rule-based parser you have written emitted confidences, and nobody was fooled by them. You assigned 0.95 to the five-digit postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. pattern and 0.6 to the "capitalized word after a comma is probably a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy." heuristic, and those numbers did one job: they sorted candidates. Nobody claimed the 0.6 rule was right on three of five inputs. It was a priority, written by hand, tuned until the ordering came out right.

A neural classifierneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s score is doing the same job. The final layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. produces a number per labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., and those numbers are pushed through a function that squashes them into a 0-to-1 range summing to 1 across the labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. at that position. It is a ranking device. The squashing function is what makes it look like a frequency, and it is the only reason anyone reads it as one.

The difference from the hand-written case is that the resemblance is close enough to be dangerous. Your 0.6 never tempted anyone to route on it. A 0.92 does.

What the number reports​

Two facts about the score are worth stating before anything else, because they narrow what a threshold on it can mean.

It is per component, not per address. There is no whole-parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. confidence field. A six-component address gives you six numbers, and reducing them to one is your decision — the usual rule being to take the weakest, since one badly read component spoils the row.

It is the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s own opinion of the chosen labelcomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., computed before the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. prior was added. So when the place index corrects a reading, the number attached to the corrected spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. is the number the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' gave that reading on its own, which is low precisely because the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' preferred something else. The gazetteer prior has the measured case: a bare Hollywood scores 0.691 as a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. without the index and 0.264 as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. with it, and the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighborhood in the hierarchy. is right. Any threshold you set will route a share of prior-corrected — that is, prior-improved — spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. to review.

The scale problem​

Set those aside and there is a third problem, which is the one calibration addresses.

Think about a weather forecaster. A good one who says "70% chance of rain" is right on about seven of every ten days they say it, and that is the entire contract — the number means what it says. A forecaster who is right nineteen times out of twenty on their "70%" days is not being careful; they are being useless in a different direction, because you cannot act on a number that understates itself either.

Measured on held-out data, the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' averages 0.913 confidence against 0.980 accuracy. It is systematically under-confident: it is right more of the time than it claims to be. The gap has a name, expected calibration error, which buckets predictions by their stated confidence and measures how far each bucket's average claim sits from its actual hit rate. On that measure the raw scoreslogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. come in at 0.0698.

Under-confidence sounds harmless and is not. A threshold at 0.9 on raw scoreslogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. cuts through the middle of a band where every parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. is correct — on twenty clean addresses the raw scoreslogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. spanned 0.828 to 0.938, and all twenty were right. You would review most of a clean file.

The correction table​

The fix does not touch the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. It is the move you make with a kitchen thermometer that reads ten degrees low: you do not re-machine the probe, you tape a card to the handle saying what each reading means.

The card is built by running the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' over data where the answer is already known, sorting the predictions by their stated confidence into twenty buckets, and recording what fraction of each bucket turned out right. That gives a translation from stated confidence to observed accuracy. The fit is isotonic, which is the one constraint that matters: the translation can never run backwards, so a raw 0.9 always maps to at least as much corrected confidence as a raw 0.8. Ordering is preserved; only the scale changes.

Applied to the same held-out data, expected calibration error goes from 0.0698 to 0.0017. The corrected number means roughly what it says. Those two figures are the held-out open-data slice; Tune confidence thresholds quotes 0.0677 to 0.0028 for the combined split that also includes in-domain rows. The open-data pair is the one to trust, because the in-domain half runs optimistic and averaging the two hides that.

It is off by default​

The table ships inside the weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. package as a small JSON file, and nothing applies it for you. The default parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. output is the raw scorelogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities., byte for byte identical to what it would be if calibration did not exist, because silently rewriting a number that consumers may have pinned thresholds against is not a change a library gets to make without saying so. A caller who wants corrected numbers builds a calibrator from the table and passes it in per call.

The practical door is Tune confidence thresholds, which has the wiring, the shipped accept-versus-precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. curve at each threshold, and the arithmetic for turning a per-spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. threshold into a per-row routing rule. That page is where to go the moment you want to act on the number rather than understand it.

What it costs​

A calibration table is fit against one modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', and modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' ship faster than fits. The table in the package was fit on an earlier modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' than the one beside it, which makes the curve a starting point rather than a measurement of what you are running. Re-fitting is repository-side work, not something a consumer can do from the installed package.

Calibration is per localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. A table fit on United States data does not transfer to French data, and a per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. table ships alongside the combined one for that reason. The combined number is an average over slices whose behavior differs, and an average is exactly the shape that hides the difference.

No threshold removes review. Even where the accepted spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. are right 99.1% of the time, that residual is real, and it is a rate rather than a set — you cannot inspect your way out of it. Budget for it instead of tuning to remove it.

Confidence is about the reading, not about the place. A high corrected score says the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' read the string the way the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. would have labeled it. It says nothing about whether the address exists, whether mail arrives there, or whether the coordinate is any good. What the model cannot do draws that boundary, and Validate an address before you use it is the check that answers the other question.