v0.6.2 step 100K eval

The v0.6.2 corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-rebalance retrain (per the street-supplement architecture) reached step 100K and was evaluated against v0.6.0 via the 2D pre-publish gate + v0-vs-neural harness. Verdict: HOLD (1 gate violation), but the picture is much better than the step-20K early evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. suggested, and v0.6.3 — pre-staged on ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. — directly addresses the only remaining regression.

Setup

• Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': v0.6.2 step 100K (model.onnx from /data/output-v062/checkpoints/step-100000)
• TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.: A100 80GB PCIe, ~16 steps/s, ~104 min wall-clock for 100K steps. No NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol..
• CorpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. changes from v0.6.0: synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 2.0 → 0.5, synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. added at weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 1.0 (122K rows: 112K US + 10K DE/GB, 35% venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.-adversarial)
• TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.: v0.6.0-a0 (unchanged)
• InferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.: admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places). fst-en-us.bin + morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition. (length-3-filtered, 1,707 canonicals from libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. street_types.txt)
• EvalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.: v0.1.2 golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. (4,561 entries), --stage3-fold flag enabled
• Gate baseline: v0.6.0 default per-tag JSON

Headline numbers

Metric	v0.6.0 baseline	v0.6.2 step 20K	v0.6.2 step 100K
Exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness.	21.1%	19.5%	22.4% ✓ (+1.3pp)
Gate violations	(ref)	3	1
dependent_locality hallucinations	0	0	0 ✓
locality recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	40.0%	39.6%	41.0% ✓
locality hallucinations	526	718	508 ✓
region recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	65.2%	67.5%	67.6% ✓ (+2.4pp)
postcode recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	76.0%	81.7%	84.1% ✓ (+8.2pp)
street recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	27.7%	22.4%	27.1% (-0.6pp — basically baseline)
house_number recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	79.0%	75.0%	74.0% ❌ (-5.0pp — only remaining violation)
country recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	24.5%	36.3%	33.1% ✓ (+8.6pp)
venue recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	29.0%	29.4%	29.3% (-)
Harness pass rateharness pass rateWhether a full address parses end-to-end with every component correct and well-segmented and no orphan spans. Stricter than label F1 — a model can win on recall yet fail the harness if boundaries slip. (415 assertions)	14.4%	13.7%	14.0% (-0.4pp)

Training curves

The val_loss and macro_f1 over trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. steps (v0.6.2 full-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. run alongside the v0.6.2b half-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. parallel). The val_loss bouncing was per-batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. noise, not a structural overconfidence pattern — macro_f1 stayed remarkably flat.

Per-tag F1 over training

Macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. hides per-tag dynamics. The trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-time per-tag F1 (computed on the val set, not the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals.) shows how each tag's quality evolved. Note that val-set F1 is consistently higher than the harsher golden-set recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. numbers in the gate report below — the val set is closer to the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. distribution.

The headline tags:

dependent_locality is 0 throughout both runs — confirming the v0.6.1 regression is fully fixed at trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time, not just at golden-evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. time. house_number stays near the 0.99 val-set ceiling (the regression to 74% shows up only on golden's harsher OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. cases).

postcode climbs steadily into the 0.98 band on both runs — matches the +8.2pp golden-set gain. locality saturates around 0.81 — matches the modest +1.0pp golden-set improvement.

What the data says

dep_locality regression: completely fixed

The whole point of v0.6.2 was to undo v0.6.1's 1066 dependent_locality hallucinations. v0.6.2 emits 0 — same as v0.6.0. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. rebalance works as designed.

Street recall recovered from the step-20K dip

At step 20K, street recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. had dropped 5.3pp (27.7% → 22.4%), which was the most alarming finding of the early-evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. gate. By step 100K, it recovered to -0.6pp (27.7% → 27.1%) — within noise. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' used the extra 80K steps to learn the contextual discrimination between streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. and venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. that contain streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-typing words.

The step-20K read was a mid-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. artifact. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' needed more steps to consolidate the discrimination introduced by the synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. counter-example trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input..

postcode and country: large structural wins

• postcode recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.: 76.0% → 84.1% (+8.2pp)
• country recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.: 24.5% → 33.1% (+8.6pp)

The country win is from synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.'s country-only template (2% of rows generating plain countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. names: "United StatesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.", "France", etc.). The postcode win is harder to explain — probably the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' freed up capacity from no longer over-emitting dep_loc on tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. that shouldn't be admin spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree..

house_number: the single remaining violation

house_number recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.: 79.0% → 74.0% (-5.0pp). DeepSeek turn 8 traced this to two mechanisms:

1. Direct: synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.'s adversarial venuesvenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. included 5th Avenue Theatre and 7th Street Bistro. These taught the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that 5th/7th (which should be house_number in real addresses) belong to venuesvenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.. (See corpus-poisoning-vulnerability.md for the broader analysis.)
2. Distributional: adding 122K rows where house_number is absent shifted the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. distribution toward "house_number is rare," so the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' under-emits the tag.

v0.6.3 (pre-staged on ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. as of 2026-05-29) ships both mechanism fixes:

• synth-no-street-v063 (the rebuilt shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.) removes 5th Avenue Theatre and 7th Street Bistro from ADVERSARIAL_VENUES. A module-load-time assertion now rejects any future digit+ordinal venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. addition.
• synth-house-venue-v063 (new shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., 32K rows) teaches house_number + venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. coexistence via patterns like "123 Main St, Sunrise Bakery, Springfield, IL 62701".

Harness: essentially tied

The v0-vs-neural harness pass rateharness pass rateWhether a full address parses end-to-end with every component correct and well-segmented and no orphan spans. Stricter than label F1 — a model can win on recall yet fail the harness if boundaries slip. moved 14.4% → 14.0%. Per the decision tree this is a sidegrade, and combined with the gate failure, the strict reading is HOLD.

But the per-tag picture shows real improvements on most tags. The harness measures strict deep-equality against the v0 rule-based parser's hand-tuned acceptance criteria — which is unforgiving on small punctuation/casing differences. The exact-match number on the golden set (+1.3pp) is a better quality signal in this case.

Gate verdict: HOLD

Per the decision tree DeepSeek turn 7 / 8 established:

Gate evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.	harness pass	Action
Pass	≥ 25%	Promote to default (HF)
Pass	15-24%	Ship as experimental
Pass	≤ 14.4%	HOLD (sidegrade not release)
Fail	any	HOLD

v0.6.2 falls in row 4: HOLD. Reasoning: the gate failure is structural (house_number regression at 5pp, well above the 2pp threshold), and even though many tags improved, shipping with a 5pp drop on a high-baseline tag would surface as a real quality regression to downstream users.

What this means for v0.6.3

v0.6.3 is the targeted fix. Its yaml v0_6_3-house-venue.yaml is on the ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume; its parquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. (part-no-street-v063.parquet, part-house-venue-v063.parquet) are too; MANIFEST is updated. Launching it took one modal run command.

Expected v0.6.3 outcome:

• house_number recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. recovers to ≥77% (the explicit target)
• dependent_locality stays at 0 (the recipe preserves what v0.6.2 fixed)
• Other tags maintain or modestly improve

If v0.6.3 passes the gate AND beats the harness baseline, it promotes. If it passes the gate but doesn't move the harness needle, it ships as experimental. If the house_number fix doesn't materialize, the linter design (see project-corpus-poisoning-vulnerability memory) becomes the priority — we need preventative checks before the next synth shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. goes in.

Reproducing

./scripts/eval-v062-checkpoint.sh 100000

Artifacts land in /tmp/v062-eval-step-100000/:

• model.onnx (117 MB, fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.)
• eval-morphology.json + .md (per-tag results)
• harness.json + .md (v0-vs-neural pass/fail per assertion)
• gate-report.md (2D gate verdict)
• verdict.txt

See also

• v0-vs-neural harness — the 14.4% baseline
• Layer 1 morphology FST eval — the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs.-only fix result that established the v0.6.x recipe
• fp32-CRF diagnostic — the precondition experiment that confirmed CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. can ship in v0.6.4
• Street-supplement architecture — the design framework
• 2026-05-28 night-2 postmortem — what started all this