Skip to main content

Eval discipline — reading the numbers carefully

Mailwoman's evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. methodology learned its most important lessons the hard way — from shipping two modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' versions that regressed on headline F1 but told a different story when the failures were examined properly. This article documents the discipline: what to measure, what not to trust, and how to read a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' release report.

Why aggregate F1 is misleading​

Per-component F1 scoresF1 scoreThe harmonic mean of precision and recall — a single number between 0 and 1 summarizing how good a classifier is at a class. F1 = 2·P·R / (P + R). are the standard metric for sequence-labelling modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. A table like this looks authoritative:

Componentv0.4.0v0.3.0Δ
countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.0.210.28−0.07
regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.0.190.18+0.01
localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.0.270.27flat
postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.0.690.76−0.07
streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.0.300.27+0.03
house_number0.790.78+0.01

The easy reading: v0.4.0 regressed on countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. and postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.. Ship the previous version.

The accurate reading, after bucketing the 1,217 postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. false-negatives and 194 countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. false-negatives into failure categories, reveals that the headline regressions are mostly evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. artifacts, not real modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' degradation.

The false-negative bucketing methodology​

For every component where F1 moves more than a few points between releases, manually inspect a sample of the disagreements between the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s prediction and the golden-set labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.. Categorize each failure into buckets. The buckets will typically fall into a few patterns:

Pattern 1: Adversarial eval entries​

The golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. contains entries chosen specifically to break the parser — multi-script addresses, ambiguous localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. names, prefix-honorific homographshomonymyOne surface, many referents: 'Georgia' is a US state in 'Atlanta, Georgia' and a country in 'Tbilisi, Georgia.' Handled by disambiguation, split across two stages — the parser resolves the tag from in-string context; the resolver late-binds the referent with geographic context.. If the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' has a known limitation (e.g., the v0.1.0 tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.'s byte-fallback on non-Latin scripts) and the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. includes entries that exercise that limitation, then F1 deltasdelta F1The F1 difference between two model versions for a tag, used to flag a regression or a gain step-over-step. on those entries are measuring whether the limitation got fixed, not whether the new weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. are better or worse.

In v0.4.0's case, 92% of the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. false-negatives were adversarial transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. entries:

بار نون وایومینگ, Wyoming, United States of America   →  pred: "yoming, United Sta"
サーモポリス, WY, United States of America              →  pred: ", WY, United State"

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' was never trained to handle these cases. The v0.4.0 weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. didn't change the behaviour on this slice — the regression was the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. holding v0.4.0 accountable for v0.3.0's known failure modes. After excluding adversarial inputs, countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. false-negatives dropped from 194 to roughly 16.

The discipline: report F1 both with and without known-adversarial slices. The "with" number is the conservative ceiling; the "without" number is the signal for whether the recipe changed anything real.

Pattern 2: Empty predictions​

When the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emits nothing for a component that the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. expects, the failure is usually a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-distribution effect — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learned a positional prior that doesn't apply to the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set.

In v0.4.0, 65% of postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. false-negatives were empty predictions on mid-position postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. like Paris 75008. The NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. downweight (the most aggressive change in the source rebalance) removed "postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-first" positional patterns from the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. mix. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learned to tag mid-position numeric tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. as house_number instead of postcode.

This is a real regression — the recipe change had an unintended side effect — but it's a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-data distribution problem, not a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' architecture problem. It suggests a targeted fix (bump NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. back up, synthesize component-order permutations) rather than a rollback.

Pattern 3: Label confusion​

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' picks the wrong labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. for a spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. In v0.4.0, 11% of postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. false-negatives were house-number confusion: 47110 Sainte-Livrade-sur-Lot, 22 Rue Jasmin → the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' predicted 22 as postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. instead of the leading 47110.

These are genuine modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' errors. They suggest the labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. is ambiguous for numeric tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. in certain positions.

Pattern 4: Span boundary slip​

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' gets the labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. right but the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. wrong. In v0.4.0, 6% of postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. false-negatives were boundary-slip cases: LE TRÉPORT, 76470 → modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' predicted ", 7647" for postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.. The tag was correct (postcode) but the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. included the preceding comma and space, and sometimes truncated the final digit.

This is a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. problem, not a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' problem — no retraining required. The fix (trimming spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. past leading/trailing non-word characters) landed in the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. without touching the model weightsmodel weightsThe learned parameters of the neural classifier, shipped as ONNX files in the @mailwoman/neural-weights-* packages. Weights are locale-specific bundles that include the model, tokenizer, and a model-card.json metadata file..

The discipline: always ask whether the failure is a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' problem or a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. problem. DecoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. fixes are cheap and don't require a retrain. Many "modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' regressions" turn out to be decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. bugs on closer inspection.

Golden-set hygiene​

The golden evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set (v0.1.2, 4,535 entries) is the single most important artifact in the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.. A few rules:

  • Adversarial entries belong in their own slice. Report F1 with and without them. The adversarial slice is a stress test, not a release gate.
  • Golden-set versions are pinned. Every evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. report references a specific golden-set version. If the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. is expanded, the old reports are not retroactively recomputed — that would falsify the historical record.
  • Annotation noiseannotation noiseErrors in human ground-truth labels (typically around 1%) that put an irreducible ceiling on eval metrics — no model can exceed the agreement of the labels it's graded against. is real. At typical 1% annotator error rates in human-labeled NER data, a 0.5–1.5pt macro_F1 shift can be noise. When a regression lands in this band, manually inspect the disagreement entries before deciding.
  • Small evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. sets amplify noise. 4,535 entries means a 1pt macro_F1 regression is ~45 flipped entries. At 1% annotator error, the false-positive rate on "regression detected" is ~10%.

Resolver eval: the name is gameable, the coordinate is not​

The discipline above grades parser spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. against a golden. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. — the stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). that turns a parsed address into a place on Earth — needs its own, and it fails differently. The trap here is grading resolution by name-match: did the resolved place carry the same name string as the gold? That question can only fail when the name is wrong, and the name is almost never wrong. There are ten US localities called "Sheldon"; "New York" is a citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., a stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., and a village 280km apart. A name-match metric scores every one of those a tie and gives the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. full marks for picking any of them.

The 2026-06-08 honest-evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. run made the cost concrete: on a leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits.-free Vermont slice the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. scored 93.7% localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. name-match while its median coordinate was 326km from the truth — it was finding the right name in the wrong stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., and name-match could not see it. Two disciplines follow.

Evaluate on geography the model never trained on​

Random evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. flatters the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. covers the same towns the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. tests, so component recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. is partly memorizationoverfittingWhen a model memorizes quirks of its training set instead of learning general patterns, so it scores well in training but poorly on new data. Guarded against with held-out evals and regularization.. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. holds specific regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. out of trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. (corpus/src/split.ts defaultHoldouts(): VT/WY/ND for US, Corse/Lozère/Creuse for FR). OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. rows in that held-out geography test the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' on places it has never seen — the only unbiased estimate of generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure.. Require a minimum slice size (we use 1,000 rows) and report it UNTRUSTED below that rather than scoring noise.

Grade by the coordinate, not the string​

Three metrics survive a name collision:

  • regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match — does the resolved regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. equal the input regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.? 100% checkable, and the single fastest way to catch a wrong-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. resolve.
  • coordinate error (great-circle, gold point to resolved centroid, p50/p90) — handles point geometry natively and exposes the wrong-instance resolves that name-match ties.
  • PIP-containment — is the gold point inside the resolved place's polygon? Name-surface-independent, but report it with a polygon-coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. denominator: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. point-geometry localities have no polygon and would otherwise count as silent failures, and tight municipal polygons reject rural addresses ascribed to the nearest town. Lead with regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match and coordinate; treat localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-PIP as a coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.-adjusted secondary.

scripts/eval/honest-eval.ts + scripts/eval/pip-containment.py implement this; see the honest-eval report for the full run.

When aggregate and functional disagree, functional wins​

The regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. fix that came out of that run looked like a pure triumph on aggregate (Vermont 326→3.4km), and the eight demo presets we read by hand caught it regressing the most famous address in the set (New York, NY → "New York Mills", 283km upstate). The aggregate had averaged that one case away. Chasing why the eight disagreed is what surfaced the deeper bug. Run the functional presets before any verdict, and treat a disagreement between them and the aggregate as a clue, not noise — the same lesson the parser side learned, on the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. side.

Oracle substitution: where the error actually lives​

Per-component F1 and coordinate error tell you how much is wrong. They don't tell you which stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). is at fault — and in a pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. (modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' → resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. → gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.) a single bad coordinate can come from a mis-tag, a missing gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. entry, or a ranking slip, and the aggregate can't tell them apart. Oracle substitutionoracle substitutionA diagnostic that hands one pipeline stage the ground-truth answer and measures the end-to-end result, to find which stage an error actually lives in. Oracle the model's tags to measure the resolver's ceiling; oracle the gazetteer hints to measure the model's. Always an optimistic upper bound. can: hand one stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). the ground-truth answer, run the rest of the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. for real, and read what error survives. Whatever is still wrong after a stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). has been given the truth cannot be that stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).'s fault — you've attributed it to everything downstream.

Which stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). you oracle decides what you learn:

  • Oracle the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s output — feed the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. the gold tags. "If parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. were perfect, how good is the coordinate?" The surviving error is the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-and-gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.'s ceiling, not the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s. This is how you answer "is this a tagging problem or a coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. problem?": hand the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. the true localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. for a row it gets wrong — if it still misses, the place isn't findable in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. or name-mismatch) and no modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' improvement can touch it; if it suddenly resolves, the tag was the bottleneck.
  • Oracle the retrieval — feed the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' perfect anchor/gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. hints. "If retrieval were perfect, how good is the parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.?" The surviving error is the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s own ceiling, independent of how good the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. is.

Run both and you've decomposed the total error across the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' and the atlas — the measurable form of the question "has the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' caught up to its database?" The gap between oracle-X and everything-perfect is X's share of the error; the gap between oracle-X and current is everyone-else's share.

Two cautions:

  • It is an upper bound, always optimistic. The oracle is perfect; the real stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). never is. "Oracle localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. → coord p50 1km" means 1km is the best the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. could do given perfect parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates., not a number you will ship.
  • Inject at the right interface. A gold localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. string (the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s output level) tests the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.; a gold resolved place-ID (skipping the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.) tests only the coordinate math. Match the injection point to the stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). you want to isolate.

The negative form of the same experiment is worth naming, because it bit us: removing a signal and watching a metric crater tells you how hard a stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). leans on that signal. Scoring an anchor-trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' with its retrieval channels absent collapsed the admin tags (countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.) to near-zero — which, read backwards, measured how essential those channels are (the v1.5.0-vs-v1.7.0 head-to-head is where that surfaced). Oracle substitutionoracle substitutionA diagnostic that hands one pipeline stage the ground-truth answer and measures the end-to-end result, to find which stage an error actually lives in. Oracle the model's tags to measure the resolver's ceiling; oracle the gazetteer hints to measure the model's. Always an optimistic upper bound. is the same idea run forwards: add a perfect signal and read the ceiling.

Verdict smokes and eval infrastructure​

A separate but related discipline surrounds trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. experiments. See VERDICT_SMOKES.md for the full framework. The evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-relevant lessons:

  • Constant-LR smokes, not cosine. Cosine decaylearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end. hides divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. under a near-zero learning ratelearning rate (LR). How big a step training takes along the gradient each update. Too high and training diverges; too low and it crawls. Mailwoman warms it up, then decays it on a cosine schedule.. A verdict smoke that uses cosine decaylearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end. will report "stable" even when the recipe would diverge under sustained peak LR. v0.4.0's cosine-LR meta-bug cost five trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. runs before it was diagnosed.
  • Full-run batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. geometry. A smoke that runs at a different effective batch sizebatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. than the full run is testing a different gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-noise regime. The smoke's "pass" verdict is not transferable.
  • Run smokes before expensive retrains. The smoke framework exists to catch divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure., NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol., and sampler starvation before they cost a full GPU run. It's the cheapest experiment in the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. loop.

The discipline checklist​

Before shipping a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' release:

  1. Report per-component F1 with and without adversarial evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. slices. The adversarial number is the conservative ceiling; the non-adversarial number is the recipe-change signal.
  2. Bucket false negatives into categories. Empty predictions, labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. confusion, spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. boundary slip, adversarial artifacts — each has a different fix and a different urgency.
  3. Distinguish modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' problemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. from decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. problemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.. SpanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. boundary slip is a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. fix; labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. confusion is a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' fix. Don't retrain for a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. bug.
  4. Inspect borderline regressions manually. A 0.5–1.5pt macro_F1 shift could be annotator noise. Look at the actual disagreements before deciding.
  5. Run verdict smokes at full-run geometry with constant LR. Cosine decaylearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end. and mismatched batch sizesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. produce false confidence.
  6. Build diagnostic tooling before you need it. corpus-audit and diagnose_regression.py were built during the v0.4.0 campaign — they would have saved most of v0.3.0's investigation time if they'd existed earlier.
  7. Grade the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. by coordinate on a leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits.-free slice. Never ship a resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. change on name-match alone — it ties same-named placesvenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. in different statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.. Evaluate on held-out geography (honest-eval.ts), lead with regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-match + coordinate error, and read the demo presets before the verdict.
  8. Score in the production SHIP-CONFIG, and oracle to attribute error. Always feed the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' the same soft channels (anchor + gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. + conventions) it ships with — an anchor-trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' scored anchor-off silently mis-reports the admin tags (countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.). When you cannot tell which stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). owns a failure, oracle-substitute one — feed it the gold answer and re-measure — and the surviving error isolates the rest.

See also​

  • Input robustness — the metamorphic INV/DIR gate in context: the three absorption layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. upstream of it and the coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. matrix
  • v0.4.0 ablation campaign retrospective — the original false-negative bucketing analysis
  • VERDICT_SMOKES.md — smoke-test discipline for trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. experiments
  • Dual-loss curvature conflict — the training divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. this methodology helped diagnose
  • Training pipeline — how corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. composition affects evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-set coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.