v0.6.3 step 100K eval

The v0.6.3 retrain (per the street-supplement architecture) reached step 100K and was evaluated. Verdict: HOLD, three gate violations, and the predicted dilution failure mode materialized.

Setup

• Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': v0.6.3 step 100K (/data/output-v063/checkpoints/step-100000)
• TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.: A100-SXM4-40GB, ~12.8 steps/s, ~130 min wall-clock. No NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol..
• CorpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. changes from v0.6.2: filtered venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. pool (removed 5th Avenue Theatre, 7th Street Bistro); ADDED synth-house-venue-v063 shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. at weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 1.0 (32K rows of house_number + venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. + streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. coexistence); dropped synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 1.0 → 0.5 (kept v0.6.2b's setting).
• TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.: v0.6.0-a0 (unchanged)
• EvalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.: v0.1.2 golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals., --stage3-fold enabled, v0.6.0 baseline

Headline numbers

Metric	v0.6.0 baseline	v0.6.2 step 100K	v0.6.3 step 100K
Exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness.	21.1%	22.4%	21.8%
Gate violations	(ref)	1	3
dependent_locality hallucinations	0	0 ✓	844 ❌
locality recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	40.0%	41.0%	34.7% ❌ (-5.3pp)
house_number recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	79.0%	74.0%	77.0% ✓ (+3pp vs v0.6.2)
street recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	27.7%	27.1%	29.2% ✓ (+1.5pp)
postcode recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	76.0%	84.1%	83.4% ✓
country recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	24.5%	33.1%	33.5% ✓
Harness pass rateharness pass rateWhether a full address parses end-to-end with every component correct and well-segmented and no orphan spans. Stricter than label F1 — a model can win on recall yet fail the harness if boundaries slip.	14.4%	14.0%	12.5% ❌

What worked

• house_number recovered by +3pp vs v0.6.2 (74.0% → 77.0%). The synth-house-venue companion shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. did exactly what it was designed to do — taught the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that house_number and venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. coexist, recovering most of the regression from v0.6.2.
• street recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. improved (27.1% → 29.2%, +1.5pp vs v0.6.2). The filtered venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. pool (no more digit+ordinal venuesvenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.) likely contributed.
• postcode, country held their v0.6.2 gains.

What broke (badly)

dependent_locality regression — predicted and confirmed

The original v0.6.1 problem came back. 844 hallucinations vs v0.6.2's 0. Rate: 2110% of expected occurrences (40 expected dep_loc spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. in the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals.; the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emitted 844 hallucinations).

This was DeepSeek's turn 10 dilution diagnosis: dropping synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 1.0 → 0.5 AND adding 32K rows of synth-house-venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. at weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 1.0 changed the anti-decompose:companion-shard ratio from v0.6.2's 1.0:0 to v0.6.3's 0.5:1.0. The anti-decompose signal got proportionally weaker. Decompose mode came back over the 20K → 100K window.

The trajectory confirms it:

• v0.6.3 step 20K: 1 dep_loc hallucination
• v0.6.3 step 100K: 844 dep_loc hallucinations

The damage accumulated gradually as the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' overfitted to certain patterns in the diluted distribution.

locality recall dropped 5.3pp

40.0% → 34.7%. Side effect of the dep_loc explosion — many tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. that should be classified as localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. are instead getting classified as dep_locality. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is confusing the two tags more than v0.6.2 did.

Harness pass rate dropped below v0.6.0

12.5% vs v0.6.0's 14.4%. v0.6.3 is a sidegrade — worse than v0.6.0 on the breadth evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.. Per the decision tree, this alone is grounds for HOLD even if the gate passed.

Gate verdict: HOLD

Per the decision tree:

Gate evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.	harness pass	Action
Pass	≥ 25%	Promote to default
Pass	15-24%	Ship as experimental
Pass	≤ 14.4%	HOLD (sidegrade)
Fail	any	HOLD

v0.6.3 falls in row 4: HOLD. Two paths from here:

1. Launch the pre-staged v0.6.4 (synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 0.5 → 0.75). DeepSeek's mechanism-3 hypothesis just got empirically confirmed by v0.6.3 going wrong in the predicted way. v0.6.4 directly counters the dilution.
2. Step back and rethink v0.6.x as a whole. Three iterations in, the pattern is clear: every corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. change produces non-trivial trade-offs. Maybe the issue is upstream of any recipe — overfittingoverfittingWhen a model memorizes quirks of its training set instead of learning general patterns, so it scores well in training but poorly on new data. Guarded against with held-out evals and regularization., evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. methodology, schema design, or modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' capacity.

The operator chose path 2. This doc captures what we learned. The v0.6.x retrospective synthesizes what the cycle revealed and proposes a rethink agenda.

Pre-staged v0.6.4 — held but not discarded

The v0.6.4 yaml + parquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. artifacts remain on the ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume. If the rethink concludes that "weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. rebalance is the right path but we just need to land the right setpoint," v0.6.4 is one modal run away. If the rethink concludes that something more structural needs to change, v0.6.4 is reference data for what would have happened.

Reproducing

./scripts/eval-v062-checkpoint.sh 100000 v063

Artifacts in /tmp/v063-eval-step-100000/.

See also

• v0.6.2 step 100K eval — the immediately-prior release
• Layer 1 morphology FST eval — established the v0.6.x recipe direction
• Street-supplement architecture — the architectural framing
• Corpus poisoning vulnerability — what the v0.6.x cycle teaches about fundamental architectural risk