v0.5.3 Diagnostic Training Review — 2026-05-27
Verdict: REVISED — v0.5.3 is the best model yet.
Initial analysis compared val_macro_f1 (0.579 vs 0.638) and concluded v0.5.3 regressed. This was wrong. The F1 numbers are not directly comparable — different tokenizerstokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses., different val sets, different labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. distributions. When tested on demo presets, v0.5.3 achieves 6/6 correct including localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Washington/regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=DC and localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=New York/regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=NY — the stubborn failures that persisted across all prior modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. The tree structure uses proper containment nesting (house_number inside streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. inside localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. inside regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.). DeepSeek's recommendation to revert was based on the same misleading F1 comparison.
v0.5.3 peaked at val_macro_f1 = 0.579 (step 28K). v0.5.1 reached 0.638. Every recipe change since v0.5.1 — wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin downweight, cosine LR, label smoothinglabel smoothingA training regularization technique: instead of putting 1.0 probability on the correct label, train to put 1 − ε on it and ε/(N−1) on each wrong label. Improves calibration; disabled in some Mailwoman runs for stability. — has regressed. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is not capacity-limited; the recipe is wrong.
Training curve
| Step | val_loss | val_macro_f1 | Note |
|---|---|---|---|
| 2,000 | 0.789 | 0.519 | WarmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup. phasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). |
| 4,000 | 0.616 | 0.543 | Rapid improvement |
| 8,000 | 0.528 | 0.558 | Slowing |
| 10,000 | 0.541 | 0.567 | val_loss uptick |
| 18,000 | 0.430 | 0.574 | Second phasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). improvement |
| 26,000 | 0.427 | 0.578 | val_loss minimum |
| 28,000 | 0.428 | 0.579 | Peak F1 |
| 34,000 | 0.435 | 0.579 | Plateau |
| 50,000 | 0.438 | 0.578 | Converged, LR at 0 |
The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' plateaued at step 18K and gained only +0.005 F1 over the next 32K steps. Cosine LR decayed to zero by step 50K, leaving no learning signal in the final third of trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input..
What went wrong: recipe comparison
| ParameterparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. | v0.5.1 (0.638) | v0.5.3 (0.579) | Impact |
|---|---|---|---|
| wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin source weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. | 2.0 | 0.3 | Primary cause. StarvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' of place-name examples. The "bare-name frequency dominance" fix removed the dominant signal entirely instead of rebalancing it. |
| label_smoothing | 0.0 | 0.05 | Contributes to lower peak F1. Smoothing prevents the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' from being confident on easy cases, which drags down macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. on high-accuracy tags. |
| LR schedulelearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end. | constant | cosine | Cosine decaylearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end. to zero killed late-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). learning. v0.5.1's constant LR allowed continued improvement through step 95K. |
| max_steps | 100K | 50K | Undertrained by 2x, but the plateau at 18K suggests more steps wouldn't help with this recipe. |
| class_weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. dep_loc/subregion | 0.3 | 1.0 | Intended fix for tag avoidance. Unclear if it helped — modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' still under-predicts these tags. |
| transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. source weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. | unspecified (default 1.0) | 0.5 | Intended to reduce non-Latin dominance. May have helped multilingual balance but didn't compensate for the wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.. |
The wof-admin downweight: diagnosis
The v0.5.2 DEMO_PRESET_DIAGNOSIS.md identified "WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. bare-name frequency dominance" as the root cause of localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. confusion. The fix was to downweight wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin from 2.0 to 0.3 — a 6.7x reduction in the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s exposure to place-name labeling.
This was the wrong fix. The wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin source provides 10M rows of countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality./dependent_locality labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. — the only source that teaches the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' "Washington" = regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., "New York" = localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.. Downweighting it starvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' of this signal. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' responded by collapsing to fewer tag types (primarily regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. and O), which is why v0.5.2 barely emits localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./house_number.
The localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. confusion was better addressed at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. time:
- QueryShape localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. prior (+2.0 bias when regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation detected) — #164
- RegionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-aware guard (skip bias when text matches regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. name) — #174
- FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. Wikipedia importanceWikipedia importanceA place-notability score derived from the count and language spread of a place's Wikipedia articles, normalized to [0, 1]. A resolver ranking prior that helps pick the prominent same-named place. (DC 0.815 > WA 0.764) — #173
These inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.-time fixes compose correctly without degrading the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s ability to emit diverse tags.
Model size is not the bottleneck
29M parametersparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. (h384, 6 layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6., 6 headsattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.) reached 0.638 on the v0.5.1 recipe. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' overfits past step 65K on this corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. — a larger modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (h512, ~68M) would overfit faster, not break through the ceiling. The bottleneck is:
- TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. recipe — the source weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. and LR schedulelearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end., not architecture
- Data diversity — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' sees 1M rows/epoch but the source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. sampler may starve it of streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label./house_number tags from structured-address sources (BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure., TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data., NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses.)
- EvalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. granularity — val_macro_f1 averages across all tags equally. A modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that perfects regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. (F1=0.99) but ignores venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. (F1=0.0) can still report 0.6+ macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric.. Per-tag F1 breakdown is needed to diagnose which tags are being sacrificed.
Recommended next steps
1. Revert to v0.5.1 recipe, retrain
source_weights:
wof-admin: 2.0 # restore from 0.3
wof-postalcode: 2.0 # restore from 1.0
ban: 3.0
tiger: 4.0
usgov-nad: 1.0
label_smoothing: 0.0 # restore from 0.05
lr_schedule: constant # restore from cosine
max_steps: 100000 # restore from 50000
Expected: val_macro_f1 ≥ 0.638 (matching v0.5.1). The v0.5.0-a1 tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. (48K vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it.) may improve or regress vs v0.5.1's effective tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. — this is the variable to watch.
2. Instrument per-tag F1 in training loop
The val evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. computes macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. but doesn't log per-tag breakdown. Add per-tag F1 to the CSV log at each evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. step:
step, ..., f1.locality, f1.region, f1.street, f1.house_number, f1.venue, f1.postcode, ...
This reveals which tags the recipe change helped and which it hurt. Without it we're optimizing a single number that hides tag-level regressions.
3. Audit per-source tag distribution
Instrument the data loader to log per-tag tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. counts per source per epoch. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' has 673M rows across 12 sources, but the source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. sampler controls which sources dominate. If wof-admin: 2.0 means the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' sees 40% bare place namestoponymA proper name for a geographic place., that explains the frequency dominance — but the fix is to add structured-address examples to wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin, not to starve the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' of place-name examples entirely.
4. Keep int8 quantization for browser deployment
The 29M modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' quantizes from 66 MB (fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.) to 17 MB (int8) with negligible accuracy lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.. This is correct for the browser cold-load budget (~60 MB total). Do not increase modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' size — the architecture is sufficient for the task. Fix the recipe, not the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'.
Demo preset results (v0.5.2 weights + inference enhancements)
model-stage2-step-001800-int8.onnx with the v0.1.0 tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.), not the v0.5.3 checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.. The inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.-time enhancements (FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. prior, QueryShape localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. guard, Wikipedia importanceWikipedia importanceA place-notability score derived from the count and language spread of a place's Wikipedia articles, normalized to [0, 1]. A resolver ranking prior that helps pick the prominent same-named place., grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model.) run on top of whatever modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is loaded. Results below may appear better than the raw modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' warrants because the priors and grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. are compensating for modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' weaknesses.Neural-only (no pipeline priors, no FST)
The raw modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' output with no inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.-time assistance:
| Preset | Result | Assessment |
|---|---|---|
| 1600 Pennsylvania Ave NW, Washington, DC 20500 | streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Pennsylvania Ave NW, house_number=1600, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=Washington, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=20500 | Missing localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Washington, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=DC. Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' assigns regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. to Washington. |
| 350 5th Ave, New York, NY 10118 | streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=5th Ave, house_number=350, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=New York, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=10118 | Missing localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=New York, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=NY. Same confusion. |
| Pier 39, San Francisco, CA 94133 | localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=San Francisco, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=CA, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Pier 39, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=94133 | Good — localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. correct. |
| 1060 W Addison St, Chicago, IL 60613 | streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=W Addison St, house_number=1060, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Chicago, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=IL, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=60613 | Correct. |
| 400 Broad St, Seattle, WA 98109 | streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Broad St, house_number=400, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Seattle, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=WA, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=98109 | Correct. |
| 90210 | postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=90210 | Correct. |
Neural-only: 4/6 correct. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' handles structured addresses well but confuses localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. on ambiguous place namestoponymA proper name for a geographic place. (Washington, New York).
Default pipeline (neural + QueryShape + FST + grouper-audit)
The full pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. with all inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.-time enhancements active:
| Preset | Result | Assessment |
|---|---|---|
| 1600 Pennsylvania Ave NW, Washington, DC 20500 | house_number=1600, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Pennsylvania Ave NW, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=NW (?), postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=20500 | Worse. Grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. injected wrong localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Pennsylvania. Neural output (regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=Washington) was overridden. |
| 350 5th Ave, New York, NY 10118 | house_number=350, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=5th Ave, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=NY, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=10118 | Lost localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=New York (grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. injected localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Ave instead). |
| Pier 39, San Francisco, CA 94133 | house_number=39, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Pier 39, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=CA, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=94133 | Lost localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=San Francisco (grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. injected localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Pier). |
| 1060 W Addison St, Chicago, IL 60613 | house_number=1060, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=W Addison St, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=IL, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=60613 | Lost localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Chicago (grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. injected localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=W Addison). |
| 400 Broad St, Seattle, WA 98109 | house_number=400, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Broad St, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=WA, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=98109 | Lost localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Seattle (grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. injected localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Broad). |
| 90210 | postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=90210 | Correct. |
Default pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.: 1/6 correct. The grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. fills the gaps the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' leaves, but the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?''s proposals are often wrong (localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Pennsylvania, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Ave, etc.) because they're structural guesses without semantic understanding.
Analysis
The neural-only path (4/6) appears to outperform the full pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. (1/6), but this is misleading. The neural-only JSON output hides unclassified spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. — decodeAsJSON only shows tags the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emitted, silently dropping all-O regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.. For example, on 1600 Pennsylvania Ave NW, Washington, DC 20500, the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' produces ONLY region=Washington and region=DC. The entire streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. + house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. + postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. (positions 0-25 and 41-46) is unclassified.
The pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. path reveals the truth: the grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. fills those gaps with provisional nodes, making the missing tags visible. The grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. is working as designed — it only fills empty spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. The problem is:
- The v0.5.2 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' barely emits non-regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. tags. On the Washington preset, 2 of ~16 tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. get typed. The rest are O. This is the tag collapse caused by the wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin downweight.
- The phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?''s structural proposals are low quality for these gaps. It proposes
LOCALITY_PHRASE("Pennsylvania")based on capitalization, not semantics. When injected at 0.55 × grouper confidence, these wrong provisional nodes appear in the output. - The JSON dedup (first-occurrence-wins) is not the issue. The grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. nodes aren't competing with neural nodes — they're filling spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' left completely empty.
The 4/6 "correct" neural-only results for v0.5.2 are correct by omission — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' gets the tags it DOES emit right but doesn't emit tags for most of the address.
v0.5.3 (step-028000, best checkpoint) — neural-only
| Preset | Result | Assessment |
|---|---|---|
| 1600 Pennsylvania Ave NW, Washington, DC 20500 | house_number=1600, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Pennsylvania Ave NW, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Washington, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=DC, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=20500 | All correct. LocalitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. confusion FIXED. |
| 350 5th Ave, New York, NY 10118 | house_number=350, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=5th Ave, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=New York, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=NY, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=10118 | All correct. |
| Pier 39, San Francisco, CA 94133 | streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Pier 39, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=San Francisco, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=CA, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=94133 | Correct. |
| 1060 W Addison St, Chicago, IL 60613 | house_number=1060, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=W Addison St, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Chicago, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=IL, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=60613 | Correct. |
| 400 Broad St, Seattle, WA 98109 | house_number=400, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=Broad St, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=Seattle, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=WA, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=98109 | Correct. |
| 90210 | postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=90210 | Correct. |
v0.5.3 neural-only: 6/6 correct. All components present with high confidence (0.90-0.97). The tree uses containment nesting: region → locality → street → house_number, which is the correct address hierarchy. This is the first modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' to produce correct localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. assignments on the Washington and New York failure cases.
Why v0.5.3 F1 is lower but results are better
The val_macro_f1 comparison (0.579 vs 0.638) was misleading because:
- Different tokenizerstokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.. v0.5.1 used v0.1.0 (24K vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it.) at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. but was evaluated against a val set tokenized with the same tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.. v0.5.3 uses v0.5.0-a1 (48K vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it.) throughout. Different subword splits produce different BIO alignmentsalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens., making F1 numbers incomparable.
- Different labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. distributions. The wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin downweight changed the val set's source mix, shifting which tags dominate the macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. average.
- Macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. hides tag-level improvements. A modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that improves localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. from 0.20 to 0.90 but regresses regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. from 0.95 to 0.85 looks like a regression in macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric.. The demo presets show the tag-level improvement directly.
Handoff for next session
Priority 1: Run eval matrix on v0.5.3 step-28000
We have 6 demo presets (all correct) but zero per-tag F1 data across 4,535 golden entries. Run the full 4-mode evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. matrix (rule-only, neural, hybrid, hybrid-joint) with the v0.5.0-a1 tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.. Compare per-tag F1 side-by-side with v0.5.1 and v0.5.2. Specifically watch localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., house_number.
Priority 2: Quantize to int8, verify size
The 48K embeddingembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding. table means the int8 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' will be ~29-30 MB (not 17 MB like v0.5.2's 24K vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'). Verify it fits the ~60 MB browser budget alongside tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. + WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. slim DB. Run demo presets on int8 to verify parity.
Priority 3: Ship v0.5.3 if eval passes
Update neural-weights-en-us/ with step-28000 int8 + v0.5.0-a1 tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. + model cardmodel cardA JSON metadata file (model-card.json) shipped with each weights bundle. It declares the model version, lineage, label set, required inference channels (anchor, gazetteer), calibration data, and training provenance.. Follow RELEASING.md pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package..
Priority 4: Verify grouper-audit is now a no-op
v0.5.3 emits tags on all spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. (no empty gaps). The grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. should have nothing to fill. Run demo presets through the full pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. and confirm the audit pass doesn't inject conflicting provisional nodes.
Priority 5: Instrument per-tag F1 in training loop
Add f1.localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., f1.regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., f1.streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., f1.house_number, f1.postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one., f1.venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. columns to the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. CSV log. Prevents the "trusted macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. across tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. versions" mistake from recurring.
Process: new eval release gate
| # | Question | Method |
|---|---|---|
| 1 | Demo presets pass? | 6 manual tests, neural-only AND full pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.. Must be 6/6. |
| 2 | Per-tag F1 improves on core tags? | Golden evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. matrix, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality./streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./house_number each ≥ previous release. |
| 3 | Overconfident-wrong doesn't regress? | Must be ≤ previous + 2pp in hybrid-joint mode. |
| 4 | No new failure-class zero-outs? | Check kryptonite and adversarial slices. |
| 5 | Int8 matches fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.? | Per-tag F1 deltasdelta F1The F1 difference between two model versions for a tag, used to flag a regression or a gain step-over-step. > 1pt = investigate. |
Never compare val_macro_f1 across tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. versions. Different tokenizerstokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. invalidate BIO alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. comparisons.
Key insight: wof-admin downweight was correct
The wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin downweight (2.0 → 0.3) didn't starve the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — it forced it to learn place namestoponymA proper name for a geographic place. from structured-address sources (BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure., TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data., NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses.) where positional context disambiguates localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. vs regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.. At weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 2.0, the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' memorized "Washington = regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality." from bare-name frequency. At 0.3, it had to attend to surrounding tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. This is the bitter lessonbitter lessonSutton's observation that general methods which scale with compute and data ultimately beat hand-engineered systems. The rationale behind Mailwoman's Ship-of-Theseus migration from rule classifiers to a learned model.: less hand-curated signal + structured context beats a strong prior that encodes the wrong pattern.
Addendum: confusion log and lessons learned
This session had significant thrashing. Documenting the confusion points so future sessions avoid them.
Mistakes made
| # | What happened | Root cause | Time wasted |
|---|---|---|---|
| 1 | Ran uncompiled TSX via npx tsx, blamed missing React import. Created PR, then reverted. | Didn't know the CLI must be run from compiled output (node mailwoman/out/cli.js). | ~30 min |
| 2 | Compared val_macro_f1 across tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. versions (0.579 vs 0.638), concluded regression. | Different tokenizerstokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. produce incomparable BIO alignmentsalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens.. No guard against this comparison. | ~2 hours of wrong-direction analysis |
| 3 | Blamed grouper-auditgrouper-auditA validation pass checking that phrase-grouper spans are internally consistent — no overlaps, no contradictions with BIO structural rules. Audit errors must be zero on a shipping model. for "overriding neural output" (pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. 1/6 vs neural 4/6). | decodeAsJSON hides all-O spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. The neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' was barely emitting tags — the audit was correctly filling gaps. | ~1 hour |
| 4 | Printed tree.roots.map(tag) and reported 4% coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.. Concluded tag collapse. | Tree uses containment nesting (region → locality → street → house_number). Only top-level roots visible without traversal. | ~30 min |
| 5 | DeepSeek recommended "do not ship, revert to v0.5.1 recipe." | Trusted the same invalid F1 comparison. Wrote 150 lines of analysis before running a single functional test. | ~1 hour of wrong-direction planning |
| 6 | DeepSeek diagnosed wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-admin downweight as "caused tag collapse." | Confirmation bias — once the F1 comparison "proved" regression, every observation confirmed it. The downweight was actually the correct fix. | Cascading effect on all subsequent analysis |
The pattern
Every confusion point shares one root cause: trusted a summary number over direct observation. val_macro_f1 over demo presets. decodeAsJSON over raw BIO stream. tree.roots over tree traversal. "29M params" over counting the embeddingembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding. table.
The fix is not more metrics. It's a mandatory step where you look at the actual output before trusting the summary. One demo preset run (30 seconds) would have prevented every error.
Process changes adopted
- Demo presets are a release gate. Run them BEFORE writing any verdict on a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run.
- Never compare F1 across tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. versions. Add
tokenizer_versionto evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. headers. - When metrics disagree with functional tests, trust the functional test. Investigate the metric.
- Print raw BIO coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present., not just JSON projection.
decodeAsJSONhides the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s gaps. - After tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. change, recheck param count. 48K vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. added 9.2M embeddingembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding. params.
- DeepSeek consultations must include functional test results alongside metrics. Aggregate metrics without functional evidence are insufficient to conclude.
Tooling improvements identified
Prospective Claude Code skills
eval-model — Demo preset release gate. Accepts modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' path + tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses., runs 6 presets through neural-only + full pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package., reports JSON + BIO coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. + source attribution. Flags regressions from baseline. Prevents the "4/6 correct but 4% coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present." blindspot that cost hours today.
training-monitor — ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. status checker. Downloads train_log.csv, parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. points, reports val_macro_f1 trajectory and best checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.. Warns on tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. version changes. Replaces the throwaway polling scripts written 5+ times this session.
wof-build — Unified WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. SQLite pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.. Chains: build-unified-wofWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. → build-importance → FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. query verification → stats report. Eliminates manual multi-step orchestration.
DeepSeek consultation improvements
Evidence checklist. The skill instructions should require that every consultation prompt includes: (1) functional test output alongside aggregate metrics, (2) tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. version when comparing modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', (3) raw BIO output not just JSON projection, (4) explicit "what changed" matrix for multi-variable comparisons.
Verify-before-concluding guard. Add a penultimate turn to the session pattern: "Before concluding — did we verify against functional tests? Do metrics and functional tests agree?" This would have prevented the "do not ship" verdict.
Empty response handling. DeepSeek returned empty responses twice today (zero-byte output files). The skill should instruct retry with a shorter prompt, and check API key validity on repeated failures.
Cross-session continuity. When the Claude conversation restarts, DeepSeek session context is lost. The skill should suggest saving key conclusions to a reference file that the next session's first DeepSeek prompt can include.
Files
- TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. config:
corpus-python/src/mailwoman_train/configs/v0_5_3-classifier-diagnostic.yaml - Train log: ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume
/output-v053/train_log.csv(1050 lines, 25 evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. points) - CheckpointscheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.: ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume
/output-v053/checkpoints/step-{002000..050000} - Best checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.: step-028000 (val_macro_f1 = 0.579126)