v0.4.0 ablation campaign

Iteration window: 2026-05-23, ~9 hours wall-clock, single Radeon 780M iGPU. Outcome: shipped a smaller subset of the planned recipe than intended; identified one process meta-bug; deferred two of three trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-side improvements to v0.4.1.

This is the technical postmortem. For the public-facing narrative, see the corresponding research blog post. For the line-by-line iteration logiteration logThe running record of each model release — the canonical 'what shipped when' for the neural classifier. entry, see PHASE_2_training.md's v0.4.0 entry.

Setup

• Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': 256-dim, 6 layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6., 4 headsattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads., intermediate 1024, max_pos 128 — 9M params total. Unchanged from v0.3.0.
• CorpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.: corpus-v0.3.0 (677M aligned rows). Unchanged from v0.3.0.
• TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.: v0.1.0 sentencepieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text.. Unchanged from v0.3.0.
• Labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it.: 21 BIO classes. Unchanged from v0.3.0.
• Hardware: gfx1103 Radeon 780M iGPU, batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=32, grad_accum=4 → effective 128, ~4.5 steps/sec sustained.

The only intended differences from v0.3.0 were:

• §1 model.crf_normalization: per_token (was implicit per_sequence) + crf_loss_weight: 1.0 (was 0.05).
• §3 model.class_weights: {O: 1.0, B-country: 2.0, ..., B-venue: 0.5, ...} (was uniform).
• §4 Source weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. rebalanced: usgov-nad: 2.0 → 1.0; wof-admin: 1.0 → 2.0; wof-postalcode: 1.0 → 2.0.

§2 (longer trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. to step 5000+ floor) and §5 (JS-side ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores. + modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'-card labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. loading) were process improvements, not lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.-surface changes. §5 had shipped before trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.. §6 was "reuse existing corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus." — non-action.

Run-by-run

Run 1 — full recipe, lr=5e-4

configs/v0_4_0.yaml. lr=5e-4 had been the v0.2.0 baseline before v0.3.0's dual-lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. instability forced it down to 1.5e-4. The §1 hypothesis was that per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. normalization would make crf_loss_weight=1.0 safe at 5e-4 again.

Diverged step 750. TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. had dropped monotonically 6.5 → 0.3 over steps 0-700, then spiked 0.3 → 3.3 over steps 700-1000. Val macro-F1 collapsed 0.36 → 0.11. Killed via train_with_resume.sh interrupt; checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. at step-500 (best so far) preserved at /data/models/checkpoints/v0_4_0/step-000500.

Run 2 — full recipe, lr=3e-4

configs/v0_4_0-lr3e4.yaml. Operator brief said "if divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. recurs by step 2000, bisect down."

Diverged step 1000. Same fingerprint. Step-750 checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. at macro_f1 0.37 preserved.

Run 3 — full recipe, lr=1.5e-4 (v0.3.0-stable LR)

configs/v0_4_0-lr1.5e4.yaml. By this point the bisect pattern was visible: 5e-4 → step 750, 3e-4 → step 1000. A pure-LR explanation predicted lr=1.5e-4 → step 1875 (linear) or ~step 2200 (proportional). It diverged at step 2000.

A factor-3.3× LR drop bought a factor-2.7× step delay. Sub-linear — confirming that LR controls when the divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. appears but isn't the root cause. The destabilizer is in the recipe, not the LR knob. Step-1250 checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. at macro_f1 0.39 preserved.

Runs 4 & 5 — ablations at lr=5e-4

configs/v0_4_0-ablate-crf.yaml (drop §1, keep §3+§4) and configs/v0_4_0-ablate-cw.yaml (drop §3, keep §1+§4). Per the issue's prescribed ablation matrix.

Both diverged at step 1000, identically. At lr=5e-4 every single-knob revert behaved like the full recipe.

Conclusion: lr=5e-4 is structurally unreachable for this codebase's dual-lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. landscape regardless of which §1/§3 knob is active. Pivot to the stable LR.

Runs 6-8 — stable-LR verdict smokes at lr=1.5e-4

Three orthogonal cells, max_steps=3000 cosine schedule:

Run	Recipe	Verdict	Peak macro_f1	Drift (last 5 evalsevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.)
6	source-only (§4 only)	PASS	0.4190	0.005
7	cw-only (§3 + §4, v0.3.0 CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. base)	PASS	0.4279	0.0036
8	crfCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.-only (§1 + §4, no class_weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.)	FAIL	n/a (train_loss=1.24 at step 3000)	n/a

cw-only looked like the clear winner — higher peak, lower drift, both under the 0.10 collapse threshold.

Run 9 — promote cw-only to full 50K

configs/v0_4_0-final.yaml, identical to cw-only but with max_steps=50000 and a fresh output_dir.

Diverged at step 2250. Same fingerprint as the full recipe at lr=1.5e-4. The smoke had been a false-positive.

Run 10 — fall back to source-only at full 50K

configs/v0_4_0-stableLR-source-only.yaml with max_steps=50000.

Survived. Peak macro_f1 0.42 at step ~2200; gradual cosine-decay decline through the rest. Step-2200 checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. became the shipped artifact: v0_4_0-stableLR-source-only/step-002200.

The meta-bug

cw-only's smoke (3000 steps, cosine schedule) had its LR back near zero by step 2750. The "pass" criterion ("macrof1 stable across the last three evalsevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. past step 2000") was measuring stability _under a decayed-to-near-zero LR. The full 50K run kept the LR near its 1.5e-4 peak for thousands of steps. That sustained-peak exposure was where the destabilization happened in every other run too.

The smoke wasn't testing the same lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. landscape as the full run. The cosine schedule's tail had been doing the heavy lifting of "stability" all along.

Process fix for future smokes:

• Constant LR for the verdict window, OR
• max_steps large enough that the cosine tail doesn't dominate. With warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup.=1000 + cosine=N, the LR is > 60% of peak roughly through step warmup + N/3. Picking max_steps so the verdict window sits in that range (e.g. 10000 keeps LR > 60% peak through step ~4300).

Math sanity-check

Before deferring §1 and §3 we audited model.py and crf.py for implementation bugs that could explain the destabilization:

• crf.py:155-164 — the per_token reduction is nll.sum() / total_tokens.clamp(min=1). Mathematically what the docstring claims. No NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.-prone path; the .clamp(min=1) guards the empty-batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. edge case.
• model.py:270-300 — class_weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. enter via nn.functional.cross_entropy(..., weight=class_weights). The PyTorch-standard path. No silent broadcast mismatch (class_weights.shape == (num_labels,) is asserted in the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' constructor).
• Dual-lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. fusion: loss = ce_loss + self.crf_loss_weight * crf_loss. Equal-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. summing at crf_loss_weight=1.0 with per_token reduction was the §1 hypothesis. The hypothesis was that per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. normalization brings CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. NLLNLL (Negative Log Likelihood). The standard form of a loss function in probability-based models: loss = −log P(correct label | input). The CRF uses the NLL of the entire label sequence. to CE magnitude. Empirically, when §1 is active, trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. destabilizes — so either per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. NLLNLL (Negative Log Likelihood). The standard form of a loss function in probability-based models: loss = −log P(correct label | input). The CRF uses the NLL of the entire label sequence. is NOT actually comparable to per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. CE on this corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., or the magnitude comparison is right but some other dynamic dominates.

No implementation bug found. The destabilization is a real recipe interaction.

Post-hoc regression diagnostic

After shipping, we ran a categorized per-tag FP/FN analysis on the shipped checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. against golden v0.1.2 (4535 entries). Tool: corpus-python/scripts/diagnose_regression.py (in tree from this iteration).

Postcode FN (1217 total)

Category	Count	Share	Pattern
empty_pred	789	65%	Paris 75008 → no postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. in output
non_latin	213	18%	バー, 47110 サント → byte-fallback noise
num_confused	136	11%	47110 SLL, 22 Rue Jasmin → predicts 22
bio_slip	73	6%	LE TRÉPORT, 76470 → ", 7647"
other	6	0.5%	—

Empty-pred dominates. §4's NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. downweight (2.0 → 1.0) removed the dominant source of "postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. comes first" patterns (NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses.'s 57M structured 911-grade rows + many FR mid-position patterns from auxiliary sources). The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' now treats mid-position numeric tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. as house_number by default.

Country FN (194 total)

178 of 194 (92%) are adversarial transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. entries — gold has English countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. names but raw input contains mixed CJK/Cyrillic/Armenian script. Examples:

بار نون وایومینگ, Wyoming, United States of America   →  pred: "yoming, United Sta"
サーモポリス, WY, United States of America              →  pred: ", WY, United State"
France, Lozère, ՍԵՆՏ-ԱԼԲԱՆ-ՍՅՈՒՐ-ԼԻՄԱՆՅՈԼ              →  pred: "" (empty)

This is v0.3.0's documented non-Latin-byte-fallback failure mode. The v0.4.0 weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. did not regress this slice; the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. suite is counting v0.3.0's known failure modes against v0.4.0. After excluding adversarials, countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. FN drops 194 → ~16.

The countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. −0.07 F1 regression is mostly a golden-set adversarial-weighting artifact, not a real recipe regression.

The decoder sidecar

The bio_slip slice (6% of postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. FN) is a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. bug, not a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' bug. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s BIO tag attribution is correct; the emitted spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. includes leading/trailing punctuation tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. Fixed in core/decoder/build-tree.ts (commit c72ab4c on main):

function trimBoundary(raw: string, start: number, end: number): { start: number; end: number } {
	let s = start,
		e = end
	const isWordChar = (i: number): boolean => /[\p{L}\p{N}]/u.test(raw[i] ?? "")
	while (s < e && !isWordChar(s)) s++
	while (e > s && !isWordChar(e - 1)) e--
	return { start: s, end: e }
}

Both node.value and node.start/node.end trim in sync so consumers slicing raw[start:end] get the same string as node.value. SpansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. that trim to empty (all-punctuation, pathological modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' output) are dropped. Word-internal punctuation (hyphens in Sainte-Livrade-sur-Lot, accents in Montréal) is preserved. 6 new boundary-trim tests pass; full repo suite (1592 tests under singleFork + 30s timeout) green.

What deferred to v0.4.1

§1 (per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. norm) and §3 (class weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.) both deferred. Both individually broke trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. in at least one tested configuration. The leading hypothesis at this iteration's boundary is that a high-variance adapter slice in corpus-v0.3.0 produces gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. spikes the per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-normalized CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. can't dampen — a corpus-audit + gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-norm probe pass per source-id would surface a candidate.

v0.4.1 threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. proposals (drafted in PR i116 body):

• Thread A — source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. tweak (NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. 1.0 → 1.5 partial restore) + synthesis pass over component-order permutations + case-norm evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.. Targets the 65% empty_pred slice. 1-2 days.
• Thread B — corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-side gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-norm probe per source-id. Find the high-variance adapter. 3-5 days.
• Thread C — verdict-smokeverdict-smokeA short diagnostic training run (≈1,500–3,000 steps) that must match the full run's gradient-noise profile, used to catch recipe instability before committing to a full 50k-step launch. framework redesign (constant-LR) + milder class_weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. ratio. 1-2 days.

Process improvements landed during the iteration

Six itemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. from the TODO.md parallel-work list shipped to main during the GPU-bound trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. windows (host-claude worked them in parallel):

Commit	What
ceb2c1f	@mailwoman/locale-gate workspace — StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2 of the runtime pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.
58eee3b	mailwoman parse --candidates <N> — Springfield-class disambiguation surface
5566cd2	corpus-audit tool — shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. distribution × source_weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. diagnostic
150c7db	runtime-pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. test hardening (AbortSignal, timing-budget, non-graceful failure)
d94261b	mailwoman parse --benchmark <N> — per-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). p50/p95/p99
c1f82ac	docs/articles/concepts/staged-pipeline-contract.md — "how to plug in a custom stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone)."

Plus the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. sidecar (c72ab4c) and the PHASE_2 iteration-log updates (6ddc170 / d499288 / 09d0d9f / 404cceb).

Numbers reference

Final evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error., shipped checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. vs v0.3.0, golden v0.1.2:

Tag	v0.4.0	v0.3.0	Δ	With adversarials excluded
countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.	0.21	0.28	−0.07	~−0.01 (nearly flat)
regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.	0.19	0.18	+0.01	similar
localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.	0.27	0.27	flat	similar
postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.	0.69	0.76	−0.07	similar (the regression is real, NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. downweight)
venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.	0.39	0.39	flat	similar
streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.	0.30	0.27	+0.03	similar
house_number	0.79	0.78	+0.01	similar (issue #57 floor held)

Macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric. raw: 0.357 vs 0.293. TokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. confidence: 0.806 vs 0.857. Full-parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness.: 0.082 vs 0.107.

Issue #116's "≥2 of 4 axes improved" metric: only fine F1 cleanly improved (streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. +0.03, house_number +0.01). Coarse F1 is mixed once adversarial denominators are excluded. Calibration flat. TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. stability negative (the central finding of the campaign).

See also

• research blog post — public-facing writeup
• PHASE_2_training.md v0.4.0 entry — canonical iteration logiteration logThe running record of each model release — the canonical 'what shipped when' for the neural classifier.
• Issue #116 — original work plan
• corpus-python/scripts/diagnose_regression.py — categorized FP/FN bucketer (in v0.4.0 branch)
• PR i116 — 10 commits with the campaign artifacts