Verdict-smoke framework

Audience

🧪 Operator documentation. For contributors designing and running trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. experiments. If you want to use Mailwoman, see Getting started.

A verdict smoke is a short (few-hundred-to-few-thousand-step) trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run whose only job is to decide whether a full-length trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run is worth launching. The full run is expensive (rented GPU, ~6–10h); the smoke runs in minutes and is supposed to surface divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure., NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol., sampler starvation, and other recipe-level bugs before they cost a full-run.

This document captures the v0.4.0 meta-bug that made the framework unreliable, and the redesigned framework v0.5.0 (and onward) uses.

Cosine-LR meta-bug

The v0.4.0 ablation campaign (issue #116, retrospective v0-4-0-ablation-campaign) ran a verdict-smokeverdict-smokeA short diagnostic training run (≈1,500–3,000 steps) that must match the full run's gradient-noise profile, used to catch recipe instability before committing to a full 50k-step launch. matrix at max_steps=3000 over the §1/§3/§4 single-knob ablations. The cw-only smoke (§3 class-weighted CE + §4 source rebalance) passed with peak macro_f1=0.4279 at step 2250 — clean curve, no warning signs.

Promoted to the full 50K run, cw-only diverged at step 2250 — macro_f1 collapsed 0.41 → 0.29.

Why the smoke missed it: the smoke ran cosine LR decay over warmup_steps=1000 → max_steps=3000. By step 2250 the cosine schedule had decayed the learning ratelearning rate (LR). How big a step training takes along the gradient each update. Too high and training diverges; too low and it crawls. Mailwoman warms it up, then decays it on a cosine schedule. to roughly 0.5 * (1 + cos(π * 0.625)) ≈ 30% of peak, and by step 2750 to ~6%. The destabilizing dynamics needed sustained peak LR to manifest, and the cosine tail collapsed the LR before the lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. curve made the divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. visible.

The smoke was reporting "this recipe is stable at this hparam slice" when what it had actually demonstrated was "this recipe is stable when LR is decaying off". Two different statements; the smoke conflated them.

This cost a 6h rented-GPU full-train cycle.

Redesigned framework

Pick one of two modes for every smoke. Default is constant-LR for any new or unstable recipe.

Mode A — Constant-LR (default)

Linear warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup. from 0 → learning_rate over warmup_steps, then flat at learning_rate for the entire smoke window. No decay.

• DivergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. shows up immediately because nothing is masking it.
• The smoke window is a "would this recipe survive at sustained peak LR" probe — which is exactly the question that matters for "will the full run blow up."
• Use this for any recipe that introduces a new lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. term, a new normalization scheme, new class weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., a tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. change, or any other lever that hasn't been validated at full LR before.

Mode B — Long-tail cosine

Cosine schedule, but max_steps >= 10000 so the cosine tail does not dominate the visible portion of the smoke window. By step 3000, with max_steps=10000, LR is still at ~84% of peak — close enough to peak that destabilization can still surface.

• Use this when the recipe is a tweak on a known-stable baseline (a small source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. nudge, a tiny class-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. adjustment within a previously-stable family) AND when the full-train cosine dynamics matter for the verdict you're trying to reach.
• Costs more wall-clock than constant-LR smokes (a 10K-step smoke is ~6× the wall-clock of a 1.5K-step constant-LR smoke).

Which to pick

Situation	Mode
New recipe (new lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss., new normalization, new tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses., new class weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.)	A — constant-LR
First time exercising a knob at a new LR	A — constant-LR
Tweak on a known-stable baseline (±20% source weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., etc.)	B — long-tail if cosine dynamics matter; else A
Pre-flight integration check (wiring, dtype, sampler) — not a verdict	Either; constant-LR is faster
Reproducing a known divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. to characterize it	A — constant-LR

When in doubt: pick constant-LR. The false-positive cost (a stable recipe gets flagged at sustained peak LR when it would actually survive cosine decaylearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end.) is small — you just promote it to the full run anyway. The false-negative cost (a divergent recipe passes the smoke and burns a full run) is what the v0.4.0 campaign paid.

How to invoke

The CLI accepts --smoke-mode {constant,long-tail} on both train and smoke subcommands.

# New recipe — default constant-LR smoke (recommended)
python -m mailwoman_train train \
    --config corpus-python/src/mailwoman_train/configs/<recipe>-smoke.yaml \
    --smoke-mode constant

# Tweak on a known-stable baseline — long-tail cosine
python -m mailwoman_train train \
    --config corpus-python/src/mailwoman_train/configs/<recipe>-smoke.yaml \
    --smoke-mode long-tail   # requires max_steps >= 10000 in the config

# End-to-end pipeline smoke (train → eval → export → quantize → package)
# Defaults to --smoke-mode constant.
python -m mailwoman_train smoke --config <recipe>-smoke.yaml

The flag overrides cfg.train.lr_schedule. Non-smoke (full) runs continue to use whatever the config declares — by default cosine, unchanged from v0.4.0.

Long-tail mode does not change max_steps; it asserts the recipe already has it sized correctly and warns when max_steps < 10000.

Reading a smoke result

A smoke is only answering "would this recipe survive sustained peak LR for the smoke window?" — it is not predicting full-run F1, full-run convergence step, or anything else.

• Lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. is finite and trending down / sideways across the full smoke window → promote to full run.
• Lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. spikes, NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol., or trends up at any point in the smoke window → recipe is unstable at this LR. Reduce LR or revisit the recipe.
• Lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. looks fine until the cosine tail (constant-LR mode disabled) → you've reproduced the v0.4.0 meta-bug. Re-run in constant-LR mode.
• Val F1 peaks early and decays in a constant-LR smoke → may still be fine in the full run (early-stopping or full-cosine recovers it), but the smoke is not the layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. that decides this. Promote and let the full run be the verdict.

Sidecars that ride alongside

Three v0.4.0 diagnostic tools that should run against every v0.5.0 smoke / full-run pair. Each is in the tree as of 2026-05-23.

• corpus-audit (corpus/scripts/audit.ts) — per-source shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.-count × source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. diagnostic. Run before trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. to verify the sampled mix matches intent. Catches the v0.3.0 "NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. = 35% effective sample" class of footgun before it costs a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. cycle.

  npx tsx corpus/scripts/audit.ts /data/corpus/versioned/<rev>/corpus-<rev> \
      --config corpus-python/src/mailwoman_train/configs/<recipe>.yaml

• diagnose_regression.py (corpus-python/scripts/diagnose_regression.py) — post-evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. FP/FN bucketing (non_latin / case_only / bio_slip / empty_pred / num_confused / other). Use it on every evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. pass, not just post-hoc — bucket distributions are how recipe choices get debugged. The v0.4.0 retrospective entry documents reference distributions for calibration.

• DecoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-trim (commit c72ab4c, core/decoder/build-tree.ts:58) — trimBoundary(raw, start, end) shrinks BIO spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. bounds past leading / trailing non-/[\p{L}\p{N}]/u characters. No retrain required; covers the bio_slip long tail the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?' (Thread E) hasn't been designed to catch yet.

Lessons added from v0.5.0 (2026-05-24)

Two hard-won additions from the v0.5.0 C-train bisect campaign. Both were footguns that the original framework didn't catch.

Smoke effective batch must match the full run

The v0.5.0 ablation-smoke at batch_size=8, grad_accum=1 (eff_batch=8) passed cleanly — lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. descended, val_macro_f1 climbed, no divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. through 1500 steps. The full train at batch_size=16, grad_accum=8 (eff_batch=128) diverged at step 800.

The recipe's stability is batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.-geometry-dependent. A smoke that doesn't reproduce the full-run's gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. noise characteristics is a smoke that can't detect this class of failure. The gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. noise at effbatchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=8 is ~4× larger per-step than at eff_batch=128 (more stochastic), which paradoxically _stabilises trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. against the curvature-conflict instability (the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' can't settle deep enough into the basin where the conflict manifests).

Rule: smoke batch_size × grad_accum_steps must equal the full-run's product. If the full run uses eff_batch=128, the smoke must too — even if that means fewer steps per wall-clock minute. The smoke's job is to reproduce the full run's dynamics, not to be fast.

Gradient-norm ratio probe before retraining

When a recipe diverges at sustained peak LR, the first diagnostic should be a gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-norm ratio probe — not another retrain with a different knob.

For a dual-lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (CE + CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. NLLNLL (Negative Log Likelihood). The standard form of a loss function in probability-based models: loss = −log P(correct label | input). The CRF uses the NLL of the entire label sequence.), load a checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. from the just-before-divergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. step. Run one forward, then two separate backwards — once with CE only, once with CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. only. Compare ‖∇_CRF‖ / ‖∇_CE‖.

# CE backward
ce_loss.backward(retain_graph=True)
ce_norm = flat_grad_norm(model)

# CRF backward
model.zero_grad()
crf_loss.backward()
crf_norm = flat_grad_norm(model)

ratio = crf_norm / max(ce_norm, 1e-12)

Reading the ratio:

• Ratio >> 1 (v0.5.0 observed 8–20×): CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. dominates. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is being pulled by the louder lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. term. Repair: reduce crf_loss_weight drastically or drop CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. NLLNLL (Negative Log Likelihood). The standard form of a loss function in probability-based models: loss = −log P(correct label | input). The CRF uses the NLL of the entire label sequence. entirely (CE-only trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. retained at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. via frozen mask).
• Ratio << 0.01: CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. has collapsed. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' has decoupled the two objectives. Repair: same — CE-only, or decouple optimisers.
• Ratio ≈ 0.5–2: lossesloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. are reasonably balanced. DivergencedivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. cause is elsewhere (capacity, data, LR schedulelearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end.).

This probe takes 5 minutes against an existing checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. and answers "which lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. is the aggressor?" more precisely than any 25-hour retrain could. Run it before spending GPU time on the next hypothesis.

Full technical write-up: docs/articles/concepts/dual-loss-curvature-conflict.md.

See also

• The knowledge ladder — why v0.4.0's failure modes mapped to missing pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. rungs (the smoke meta-bug is the process-side companion).
• Phase 8 — v0.5.0 fresh-slate — Thread F lands this framework before B/C/E start trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input..
• Phase 2 — training iteration logiteration logThe running record of each model release — the canonical 'what shipped when' for the neural classifier. — v0.4.0 entry has the original false-positive case.
• v0.4.0 ablation campaign retrospective — full incident write-up.
• Dual-loss curvature conflict — the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.-as-aggressor finding + cooperative-vs-conflict regime modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'.
• Operations — working norms; smokes commit at the unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. boundary like any other change.