v0.8.1 — The MLM decisive round

The question

Does masked-language-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. of the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). improve the supervised parser at the task ceiling? The 40k round (lr 5e-5 cosine) showed pretraining's secondary gains (calibration +2.0pp, harness +4.8pp) but BOTH arms under-trained the product metric (resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. 48–50% vs shipped 96.1%) — a broken recipe, not an init verdict. This round re-runs the A/B at the v0.7.2-proven recipe so the comparison happens where the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' can actually reach ~96% resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon..

What changed (single variable)

Both arms are identical except the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). initialization:

knob	both arms
learning ratelearning rate (LR). How big a step training takes along the gradient each update. Too high and training diverges; too low and it crawls. Mailwoman warms it up, then decays it on a cosine schedule.	1.5e-4 constant (v0.7.2-proven)
steps / warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup.	100,000 / 1,000
label smoothinglabel smoothingA training regularization technique: instead of putting 1.0 probability on the correct label, train to put 1 − ε on it and ε/(N−1) on each wrong label. Improves calibration; disabled in some Mailwoman runs for stability.	0.1 (both arms — keeps calibration fair)
CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.	off (crf_loss_weight=0.0)
precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. / seed	bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. / 42
corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. / tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.	v0.4.0 / v0.6.0-a0 → A/B F1 comparison is valid

arm	encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). init
A pretrained	init_from output-v080-mlm-pretrain/checkpointscheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang./step-020000 (MLM)
B scratch	fresh random init

TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. + corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. are held constant, so the "never compare F1 across tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. versions" rule is not in play — this is a clean single-variable comparison.

Held-out metrics

Three metrics, auto-parsed from each runner's own JSON/MD sidecar (evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. figures are never hand-typed — see #211/#212):

1. resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. — the PRODUCT metric, on real OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. points (non-circular). Primary decision metric.
2. harness v0-neural pass-rate — the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-lineage regression gate (secondary).
3. calibration — fraction of WRONG predictions emitted at confidence ≥ 0.9 (lower is better; the original pathology was ~81% at ≥0.9).

Kill-point rule (DeepSeek turn-4 consult, 2026-05-31)

• pretrained does not ≥ scratch resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. (within ±0.5pp) and harness+calibration gains disappear/reverse → DROP pretraining (v0.7.2 stays default).
• pretrained matches/slightly beats scratch resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. (even +0.3pp) and retains the calibration/harness gains → SHIP with pretraining.
• pretrained beats scratch resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. by ≥ 1pp → SCALE pretrain 20k → 100k and re-run.

Results

Generated by scripts/eval-v081-decisive.sh (resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. re-run at 10k OA); figures are auto-parsed from each runner's sidecar, never hand-typed.

metric	A pretrained	B scratch	Δ (A−B)	v0.7.2 ref
resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. (10k OA)	97.3% (9730/10000)	97.5% (9750/10000)	−0.20 (0.9σ — tie)	96.1%
harness pass-rate	19.04% (79/415)	20.72% (86/415)	−1.69	19.5%
wrong-pred confidence p90 (cal.)	0.949	0.949	0.00	—

Reference column = shipped v0.7.2, cited from 2026-05-30-v0.7.2-eval.md (resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. 96.1%, harness 19.5%).

ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. was re-run at the full 10,000-row OA sample (the auto-runner's 3k first pass showed −0.60pp, within noise; 10k tightens it to −0.20pp / 0.9σ — a statistical tie). Both arms use the post-#222 gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-alias matching, and both clear shipped v0.7.2 (96.1%) and v0/PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. (95.8%) on real points.

Calibration note: the metric here is the 90th-percentile confidence of WRONG predictions (lower = better calibrated), identical for both arms (0.949). It is NOT the "fraction of wrong preds at ≥0.9" the auto-runner's column labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. implied — that labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. conflated a percentile and has been corrected. Either way the init-attributable calibration delta is zero.

Reading

At the task ceiling, MLM pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. of the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). buys nothing:

• Product metric (resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.): a tie (−0.20pp, 0.9σ). Pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. is neutral, not harmful.
• Harness: scratch is ahead by 7 assertions (−1.69pp) — small, but the wrong direction for pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input..
• Calibration: flat (identical wrong-pred confidence distribution).

The decisive point is the disappearance of the 40k round's gains. That under-trained round (lr 5e-5, 40k) had shown pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. +4.8pp harness and a calibration edge — which is what kept the track alive. Re-run at the v0.7.2-proven recipe where the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' actually reaches the ceiling (Arm A's step-28k evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. was already v0.7.2-class: localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. 0.84 / regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. 0.75 / streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. 0.94 / house_number 0.99 / postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. 0.999), those gains vanished. They were an under-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. artifact: a from-scratch encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). simply hadn't caught up yet at 40k, so the pre-trained init looked better. Given enough steps, scratch catches up and the gap closes.

Decision

DROP pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.. v0.7.2 stays the default; the MLM track is closed as a clean negative result. This is exactly DeepSeek's pre-registered kill-point (turn-4): the SHIP branch required pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. to retain the calibration/harness gains, and at ceiling it does not. The product metric being a tie (not a lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.) means the honest conclusion is "no benefit," not "actively worse" — but a +1 trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). (20k MLM pre-train) and the attendant pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. complexity are not worth a statistical tie plus a slightly-worse harness.

What we keep from the track: the MLM pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. code (pretrain.py, masking.py, forward_mlm) lands cleanly and is reusable if a future, larger modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' or a different objective (e.g. ELECTRA-style, or pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. on a much larger unlabeled corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.) revisits it — the negative result is specific to this 29M-param encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters)., 20k-step pre-train, en-us corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. The forward prize remains resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. depth (see the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. failure analysis + backlog), where the product-metric gains have been real.