Methodology — falsification-first, gate-promoted

Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' work tempts you toward vibes. A metric moved, the demo feels better, ship it. Every team says they resist this; the resistance only holds if it's built into the machinery rather than the mood. This article describes the machinery: how a change moves from hypothesis to shipped, and why the same rules that keep the work rigorous are the ones that keep it open.

One sentence, if you want it portable: we run the scientific method on an engineering budget. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is ~29M parametersparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., so the full loop — hypothesis, probe, retrain, gate, ship or falsify — fits inside an evening. Cheap inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. is the advertised benefit of small modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'; cheap rigor is the underrated one.

Hypothesis to shipped

Every change starts as a hypothesis with a named, measurable prediction, and the first move is the cheapest test that could kill it. A 30-minute GPU run waits behind a 2-minute probe; a retrain waits behind a data-side lever; a lever waits behind a diagnostic that characterizes the failure. Most hypotheses die at the probe stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone)., which is the point — the expensive step runs only on survivors.

A concrete week (July 2026, all in the public tracker): French accented streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. names were losing their diacriticsdiacriticAn accent mark that modifies a letter (é, ñ, ç). Address normalization must fold diacritics for matching without discarding the information a user typed. at the tokenizer. The cheap fix — splicing merged French piecesECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). into the vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. with mean-initialized embeddingsembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding., a trick that had worked twice for other languages — was tried first, measured, and made things worse, for a reason worth knowing: the isolated é piece carried a strongly-trained signal, and averaging embeddingsembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding. inherited its confidence. That falsification, with its mechanism, became the spec for the fix that worked: a 12,000-step fine-tune on the spliced tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.. Thirty-three minutes on one A100. Falsified, respecified, retrained, gated, and shipped between dinner and sunrise.

Ablation with a control group

An ablation tells you what a component contributes. Ours carry an extra obligation: every change must prove what it didn't touch. When a lever targets French, the untouched panels — US, German, the gauntlet's regression cases — must come back byte-identical, not "roughly flat." The null hypothesis is verified, never assumed.

The same discipline pins the variable under test. The fine-tune above held the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. byte-for-byte constant so the spliced tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. was the only moving part; when the result improved, there was exactly one candidate explanation. Single-variable arms plus byte-stable controls: ablation, with the control group most ablation studies skip.

Per-locale floors

CoveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. grows one localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. at a time, and each localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. gets its own floor — a panel and a threshold that later changes must not sink. A lever that lifts French while taxing English would pass any aggregate metric; it cannot pass a per-locale gatelocale gateStage 2 of the runtime pipeline: rule-based locale detection from the query shape's script and known-format signals. Returns a LocaleHint with the top candidate and alternatives, surfacing disagreement with an explicit --locale flag.. This is what lets the system take on new address conventions without silently spending the quality budget of the old ones. (For why aggregates mislead in this domain specifically, see Eval discipline.)

Gates that can't drift

The failure mode of self-administered rigor is the goalpost that moves between runs — the re-derived baseline, the softened threshold, the deleted test. Three structural rules close it:

• Baselines are pinned artifacts. A baseline is recorded with its checksum and the verbatim command that produced it, at creation time. Comparing against a re-derived baseline under a different tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. or modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. is a category error, and the harness refuses it.
• Expected failures are tracked, not hidden. A known-bad case is an xfail with an issue reference — it stays visible in every run's output. Deleting a failing case and tracking it are different acts, and the diff shows which one happened.
• The regression suite is an executable bug log. Every bug ever fixed remains a permanent, named test with its coordinates and tolerances. A fixed bug that un-fixes fails the gate — loudly, before release, as one did the night before v5.6.0 shipped, when a data rebuild re-broke a namesake case the suite had guarded for weeks.

Promotion runs through three layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6.: the regression log, metamorphic invariants (uppercasing an address, adding a trailing dot, or a one-letter typo must not move the pin — properties that need no answer key and so can't be overfit), and a held-out draw where candidate and production modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' compete on addresses neither has seen, judged by a z-test. The held-out layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. wins every conflict with a prettier metric.

Negative results are deliverables

A falsified hypothesis with a mechanism is treated as shipped work — written up in the issue tracker with the measurement that killed it and the reason it failed, at the same standard as a featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them.. This is partly principle and partly economics: the falsification record is what stops the same dead lever from being re-run six months later, and in practice the mechanism inside a failure is usually the specification for the fix. The tracker holds weeks where the falsified hypotheses outnumbered the shipped featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them., and those were good weeks.

Openness as the enforcement mechanism

Keeping research open while keeping it rigorous is usually framed as a tension. We've found the opposite: the public record is the rigor. A published baseline can't be replaced without a visible diff, and neither can an xfail. A falsification posted with its mechanism stays findable when the retry gets tempting. Every claim in the docs traces to an evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. whose harness is in the repo, which means anyone — including us, six months later — can check whether the number still holds.

The disciplines above would be worth keeping in a private codebase. In an open one, they stop being self-administered: the audience is the control group.