Isotonic confidence calibration — neural-weights-en-us v4.4.0

Post-hoc calibration of the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs.'s per-spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. softmaxsoftmaxThe function that converts a vector of logits into a probability distribution summing to 1, applied after priors and biases are added to the emission logits. confidence (the conf= a resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. or human reads off the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.). Method: isotonic regressionisotonic calibrationA post-hoc calibration that fits a monotonic map from raw model scores to true probabilities without retraining (via PAVA). Mailwoman's confidence calibrator. (PAVA) over (raw confidence, correct?) pairs from a 50/50 OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. + trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. calibration set. Fit on 80%, every number below measured on the held-out 20%. Task #59 (#240 PR3).

correct? is a normalized exact-or-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-subset spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. match (so streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. decomposition and multi-word fragmentation aren't penalized), so the absolute accuracy runs mildly optimistic — isotonic corrects the reliability shape, which the lenient threshold leaves intact. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. half is in-domain (the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' trained on it); the OA-only row above is the trustworthy held-out ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated)..

Headline

Split	ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). raw	ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). calibrated	target
Combined (deliverable)	0.0643	0.0034	<0.05
OA-only (held-out, trustworthy)	0.0593	0.0113	—
corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-only (in-domain)	0.0682	0.0051	—

MCE (bins n≥20) 0.3303 → 0.1338 · Brier 0.0278 → 0.0224 · n_fit=28556 n_eval=7139 spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree..

MCE is reported over bins with ≥20 samples. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is confident — ~94% of held-out spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. sit in [0.93, 1.0] — so equal-width bins below ~0.7 hold a handful of samples each and their all-bins max gap is single-sample noise, not a calibration failure. ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). (sample-weighted) is the headline; it weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. each bin by its mass.

Reliability (held-out eval, raw confidence)

confidence bin	n	mean conf	accuracy	gap
[0.00, 0.07)	5	0.006	0.000	0.006
[0.07, 0.13)	2	0.093	0.500	0.407
[0.13, 0.20)	1	0.188	1.000	0.812
[0.20, 0.27)	2	0.246	1.000	0.754
[0.27, 0.33)	4	0.305	1.000	0.695
[0.33, 0.40)	8	0.380	0.375	0.005
[0.40, 0.47)	18	0.436	0.667	0.231
[0.47, 0.53)	35	0.498	0.829	0.330
[0.53, 0.60)	31	0.567	0.774	0.207
[0.60, 0.67)	46	0.640	0.891	0.251
[0.67, 0.73)	59	0.703	0.915	0.212
[0.73, 0.80)	137	0.769	0.912	0.144
[0.80, 0.87)	288	0.838	0.858	0.019
[0.87, 0.93)	3843	0.916	0.986	0.070
[0.93, 1.00)	2660	0.946	0.988	0.042

Reliability (held-out eval, calibrated confidence)

confidence bin	n	mean cal	accuracy	gap
[0.00, 0.07)	2	0.024	0.000	0.024
[0.07, 0.13)	1	0.117	0.000	0.117
[0.20, 0.27)	1	0.204	0.000	0.204
[0.33, 0.40)	1	0.333	0.000	0.333
[0.47, 0.53)	10	0.499	0.900	0.401
[0.60, 0.67)	1	0.623	1.000	0.377
[0.67, 0.73)	27	0.726	0.593	0.134
[0.73, 0.80)	24	0.738	0.833	0.095
[0.80, 0.87)	436	0.853	0.869	0.016
[0.87, 0.93)	153	0.908	0.895	0.012
[0.93, 1.00)	6483	0.988	0.987	0.001

ECE by locale (held-out eval, raw → calibrated)

localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.	n	accuracy	ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). raw	ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). calibrated
NL	188	0.979	0.1520	0.0883
DE	189	0.852	0.1276	0.1317
FR	803	0.968	0.0734	0.0265
US	5959	0.980	0.0671	0.0079

ECE by tag (held-out eval, raw → calibrated)

tag	n	accuracy	ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). raw	ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). calibrated
streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.	792	0.991	0.0980	0.0306
postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.	1582	0.975	0.0919	0.0248
venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.	411	0.981	0.0838	0.0218
regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.	1219	0.993	0.0733	0.0138
street_suffix	591	1.000	0.0703	0.0167
localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.	1618	0.974	0.0526	0.0220
house_number	768	0.911	0.0216	0.0552

Abstention curve (calibrated confidence)

Accept spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. at or above the threshold; route the rest to review. PrecisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. is the accuracy of the accepted set.

threshold	coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. (accepted)	precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.	reviewed
0.50	99.9%	97.59%	0.1%
0.80	99.1%	97.78%	0.9%
0.90	92.7%	98.56%	7.3%
0.95	86.4%	98.82%	13.6%
0.97	81.6%	98.92%	18.4%

The single global table is fit across all localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for./tags, so it under-serves the worst-calibrated subgroups — the per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. rows show where the one-size table leaves residual error (the OOD localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. and rare tags run far higher than the US/FR-dominated global ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated).). A per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. table is the natural next step once the deployed multi-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is the calibration target (#368).

20-bin lookup table (raw → calibrated)

bin center	calibrated
0.025	0.475
0.075	0.487
0.125	0.487
0.175	0.487
0.225	0.487
0.275	0.490
0.325	0.526
0.375	0.667
0.425	0.731
0.475	0.738
0.525	0.800
0.575	0.854
0.625	0.854
0.675	0.854
0.725	0.854
0.775	0.854
0.825	0.854
0.875	0.938
0.925	0.992
0.975	0.992

How it's wired

The table ships as data/eval/calibration/isotonic-en-us-v4.0.0.json and is turned into a (raw)=>calibrated function by the OPT-IN decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. calibrator (core/decoder/calibration.ts → createCalibrator). Default parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. output is unchanged (byte-stable); pass the calibrator via ParseOpts.calibrate / BuildTreeOpts.calibrate to emit calibrated conf=. Regenerate with scripts/eval/{build-calibration-set.py,collect-span-confidences.ts,fit-isotonic-calibration.py}.