Confidence calibration

Every spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. the parser emits carries a number: conf="0.94" on the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., conf="0.81" on the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.. You're going to want to do something with that number. Auto-accept the parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. you're sure of, route the shaky ones to a human, lower a resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.'s trust when the parser is hedging. Before you can lean on it, there's one question worth answering: when the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' says it's 94% sure, is it right 94% of the time?

For a long while, ours wasn't.

The forecaster's promise

Think about a weather forecaster. A good one who says "70% chance of rain" should be right on about seven of every ten days they say it. That's the whole contract: the number means what it says. If it rains on nineteen of every twenty "70%" days, the forecaster is sandbagging; if it rains on half, they're overselling. Either way you stop trusting the percentage, and the percentage was the useful part.

A neural classifierneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' makes the same promise implicitly, every time it hands you a softmaxsoftmaxThe function that converts a vector of logits into a probability distribution summing to 1, applied after priors and biases are added to the emission logits. probability. And like a lot of cross-entropycross-entropyThe standard classification loss: it penalizes a model for putting low probability on the correct label. Per-token negative log-likelihood is the cross-entropy of each token's label.-trained modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', ours broke it: confident in some bands, timid in others, with no single knob that lined the number up with reality.

We measured the gap with Expected Calibration Error. Bucket the predictions by stated confidence, and in each bucket compare the average confidence to how often those predictions were actually right. Average the gaps, weighted by how many predictions land in each bucket. Zero is a forecaster you can trust. The shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' sat around 0.060, about six points of slack between what it claimed and what it delivered — and, interestingly, mostly under-stated: a spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. it called 92% was right closer to 98% of the time. Slack is slack in either direction; a number you can't trust to be too low is no more useful than one you can't trust to be too high.

A correction sticker, not a new model

The fix doesn't touch the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. It's the same move you'd make with a cheap kitchen thermometer that reads ten degrees low. You don't crack it open and re-machine the probe; you tape a little card to the handle that says "reads 340, it is 350."

We build that card by running the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' over a pile of addresses where we already know the answer, watching where its stated confidence drifts from how often it turns out correct, and fitting a lookup table that translates raw confidence into a calibrated one. The fit is isotonic, which is a fancy word for a simple promise: the translation never runs backwards, so a raw 0.9 always maps to at least as much corrected confidence as a raw 0.8. Twenty bins, monotone, nothing exotic. And we calibrate the number you actually read, the per-spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. conf= aggregated across a spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.'s tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., rather than some internal per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. probability you'd never see.

The result, measured on addresses the fit never saw: ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). 0.060 → 0.0055, an eleven-fold tightening. The Brier score, another calibration measure, drops from 0.029 to 0.024. The conf= now means roughly what it says.

Here it is, drawn live from the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s own held-out reliability — the left panel is the promise, the right is what you do with it:

Loading calibration data…

On the left, each dot is a confidence bin; its height is how often those spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. were actually right, and a perfectly calibrated forecaster sits on the dashed diagonal. The amber dots — raw softmaxsoftmaxThe function that converts a vector of logits into a probability distribution summing to 1, applied after priors and biases are added to the emission logits. — float above the line in the middle bands (the under-statement), then the teal dots settle back onto it. On the right is the move that makes calibration pay rent: raise the auto-accept bar and the precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. of what you keep climbs while coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. falls, so you can pick a point — accept above 0.90, say — and know you're auto-handling ~97% of spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. at ~98% precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1., routing the rest to review.

Why the number doesn't flatter itself

Two details keep this from being a number that flatters itself.

First, we fit on one slice of data and report on another. Grading a calibrator on the same examples you tuned it on is like a forecaster grading last week's forecasts after he's already seen the weather; the score comes out better than the future will. So every number above is measured on a held-out 20% the fit never touched.

Second, the calibration set is half OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. (real, government-sourced addresses the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' never trained on) and half trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. rows. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. half is in-domain, so the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' runs a touch optimistic there; the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. half is the held-out, real-world signal. We report the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries.-only number next to the combined one (0.056 → 0.019) precisely so that optimism stays visible instead of getting averaged away — the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries.-only figure is three times the combined one, which is exactly the kind of thing a single averaged headline would hide. Trust the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. number more. We'd rather hand it to you than bury it.

It's off by default

Calibration is opt-in. The default parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. output is byte-for-byte what it was before, because silently rewriting the conf= values would break anything downstream that pinned to them. So the lookup table ships as a small JSON, and a caller who wants calibrated confidence builds a calibrator and passes it in:

import { createCalibrator } from "@mailwoman/core/decoder"
import table from "@mailwoman/neural-weights-en-us/calibration.json"

const calibrate = createCalibrator(table)
const tree = await classifier.parse(input, { calibrate })

Leave it out and you get today's numbers, unchanged.

Why bother

A calibrated conf= is the difference between "the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' felt good about this" and "this is right about 94% of the time." The first is a vibe. The second is something you can build a threshold on, accept-above-0.95 and review-the-rest, and have the threshold mean what you set it to. That's the entire reason to put a number on the output: so you can act on it without re-deriving the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s mood every time.

The pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. that produced the table (build the set, score the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', fit the curve) lives in scripts/eval/, and re-running it for a new modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is three commands. The full reliability tables and methodology are in the v4.13.0 calibration eval report; the original write-up covers the first fit.

The same lever, one level up

Everything above is a single spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.'s conf=. The same move works on the whole answer. mailwoman resolves a parse to a coordinate, and that coordinate carries a confidence from the components that produced it, so you can put the threshold on the result, not just the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. On a messy held-out set — lowercased, abbreviated, postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. dropped — graded by whether the resolved point lands within 25 km of truth, the dial behaves the same way: 84% of answers are right at full coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present., climbing to 97% as you keep only the confident ones and let recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. fall to the sure cases. The discrimination holds on a slice the curve was not drawn on. A search index hands back one best guess with no dial; that's the part it can't give you. The full sweep, the held-out check, and the slices where it loses are in the precision-lever report.