2026-07-17 — M1: the stack ablation. The atlas channel is the stack; two repairs are inert and two are negative.
The road-to-v7 Track M opener, and the external audit's central unanswered question: how much of
the shipped quality is the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', and how much is the stack of prior channels and repair passes
around it? Eleven configurations — the full production path, each lever off one at a time, the
gate-battery config, and the raw modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' alone — scored on four arenasarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. with the gate battery's own
metric (decodeAsJSON → fold → per-tag exact-match F1; runner scratchpad/m1-ablation.run.ts,
results scratchpad/m1-ablation-results.json). Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' = shipped v381/v6.5.0, en-us weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., all
sweeps CPU-only.
Reading note: ablating a channel the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' was TRAINED with is deliberate out-of-distribution input (#566/#685), so a channel delta reads "what this channel is worth to the trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'" — not "what a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' trained without it would score." That is the right question for the consolidation ledger (P2): it prices what each layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. earns at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. time today.
Micro-F1 deltas vs the full production config
| config | golden-us (2660) | golden-fr (1546) | adversarial (49) | parity (376) |
|---|---|---|---|---|
| full (production) | 85.5 | 91.6 | 90.1 | 69.6 |
| battery (gate config) | +2.3 | −0.0 | −0.4 | +0.1 |
| no-heal | −0.6 | −0.8 | +0.2 | −2.6 |
| no-pcrepair | 0.0 | −0.0 | 0.0 | +0.1 |
| no-queryshape | +2.3 | 0.0 | −0.4 | −0.0 |
| no-anchor | +0.8 | −1.6 | 0.0 | −0.3 |
| no-gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. | −10.4 | −1.7 | −0.8 | −3.9 |
| no-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. | −0.1 | +0.3 | +0.2 | −0.1 |
| no-conventions | 0.0 | 0.0 | 0.0 | +0.2 |
| no-suppress | +0.6 | 0.0 | −0.2 | −0.1 |
| raw (modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' alone) | −8.8 | −9.2 | −0.7 | −8.1 |
The four findings
1. The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. channel IS the stack. Removing it costs −10.4 micro on golden-us and collapses the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. tag everywhere (−66.5 us / −60.0 fr / −63.4 parity — countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. is carried almost entirely by the atlas feed), with localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. −21.6 and regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. −16.6 on US riding with it. The raw-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' rows confirm it: of the ~9-point total stack value on us/fr/parity, nearly all of it is this one channel. The audit's "retrieval-augmented" framing is literal — the atlas channel is load-bearing; everything else is trim.
2. The query-shape prior COSTS 2.3 micro on clean US addresses — and this is the clean-arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.
drift, diagnosed. no-queryshape reproduces the battery numbers exactly: the entire
production-vs-battery gap on golden-us is the query-shape prior, and it is driven by localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.
−7.8 under the prior. On FR, adversarial, and parity it is ~0. Two consequences: (a) the gate
battery (which never feeds queryShape) has been over-reporting the production config by 2.3 micro
on US golden — the gate measures a config production does not run; (b) the prior itself misfires
on clean full addresses, the exact "libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. clean-arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. drift" the audit told us to watch.
Follow-up filed: either fix the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. interaction or make the gate score the production config.
3. Two repairs are inert; one is mildly negative. postcodeRepair measures 0.0 on every arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.
(first time it has ever been measured — the battery never exercised it); the conventions mask is
0.0 everywhere; and suppressGazetteerNearPostcode (the #956-era near-postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. choreography) is
+0.6 on golden-us when removed (house_number +1.9, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. +1.2) — a repair built for an older
modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' era that now slightly hurts the current one. All three are P2 consolidation-ledger
candidates: retire, or show a arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. where they earn their keep.
4. The word-consistency heal and the postcode anchoranchor inferenceA technique where structured knowledge (postcode locations, gazetteer place names) is injected into the model as soft input features — not as deterministic overrides. The model still decides the final labels, but the anchor signal biases it toward correct admin tags. earn their keep — asymmetrically. The heal is worth +0.6/+0.8/+2.6 (us/fr/parity — largest exactly where input is hostile; keep, default-ON vindicated). The anchor channel is worth +1.6 on FR (house_number +5.7 — FR leans on it) but reads −0.8 on golden-us, i.e. mildly counterproductive on clean US input. Not actionable alone, but it says channel value is localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.-asymmetric, which the per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. capability manifest should eventually reflect.
Cross-check
The battery arm scores golden-us 87.8 vs the promotion gategated promotionA release discipline where a model ships to production only if it passes pre-registered metrics (e.g. 'DE locality ≥ 70%, no tag regresses > 2pp'). Failing models can be uploaded as experiments but not promoted to default.'s 86.8 for the same weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. — a
1.0-point loader difference (createScorer explicit-paths vs loadFromWeightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. package feed) that is
constant across configs and does not affect any delta above.
What this feeds
- P2 (repair-stack consolidation ledger): pcrepair, conventions, suppress enter with measured deltas ≤0 — each must show a justifying arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. or be retired at the next consolidation retrain.
- The queryShape–localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. interaction is a new, quantified defect (−7.8 localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. on clean US)
with a one-line repro (
scratchpad/m1-ablation.run.ts, configsfullvsno-queryshape). - The audit disposition: Kimi's "repair-stack weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values." and "prior-ablation matrix" itemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. are now RUN; the clean-arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.-drift watch item has a diagnosis.