Layer 1 (street-morphology FST) eval — 2026-05-28

Applied the newly-landed street-morphology FST as a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs.-only fix on the v0.6.1 weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. and re-ran the 4561-entry golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals.. Goal: measure whether the morphology prior alone suppresses v0.6.1's 1066 dependent_locality hallucinations without retraining.

TL;DR: It does not. The mechanism is structurally correct (dep_loc hallucinations drop monotonically as the penalty is strengthened) but the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s overconfidence on synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-induced predictions is too high for any practical decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs.-time bias to flip. The right deployment of LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 1 is alongside a v0.6.2 retrain that adds O-tagged streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. slots to the negative-example corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus..

Setup

• Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': model-v061-step-100000-int8.onnx (the experimental v0.6.1 weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. with synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 2.0, that produced the original 1066 dep_loc regression)
• TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.: v0.6.0-a0 (multi-script; matches the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.)
• Admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places).: fst-en-us.bin (the production admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places).)
• Morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition.: built in-process from the 60 libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. street_types.txt dictionaries (core/data/libpostal/dictionaries/{locale}/street_types.txt)
• Golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals.: data/eval/golden/v0.1.2/ — 4561 entries (US + FR + adversarial)
• Script: scripts/eval-morphology-fst.ts (new, takes explicit weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. + FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. paths)

Results

Configuration	dep_loc hall.	street_suffix hall.	street_prefix hall.	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness.
v0.6.1 neural-only (on record)	1066	198	31	31.1%	27.5%	18.8%
v0.6.1 + admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places). only	1063	204	32	31.2%	27.5%	18.8%
+ morphology, defaults	1050	332 ❌	40	31.2%	26.2%	18.2% ❌
+ morphology, length≥3 filter	1058	238	32	31.2%	27.1%	—
+ morphology, low bias + −6.0 pen	1044	213	32	31.3%	27.5%	—

Defaults: maxAffixBias=3.0, maxNeighbourStreetBias=2.0, dependentLocalityPenalty=2.0. Tuned: maxAffixBias=1.0, maxNeighbourStreetBias=1.0, dependentLocalityPenalty=6.0.

Findings

Admin FST does nothing for street-side errors

v0.6.1 neural-only vs v0.6.1 + admin FST only is essentially a no-op (1066 vs 1063 dep_loc hallucinations). Confirms the structural diagnosis from the WOF hierarchy gap doc: the admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places). has no streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. placetypesplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. to match, so it can provide no negative-evidence anchor for streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. Synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. exploited exactly this vacuum.

Layer 1 mechanism is sound but its magnitude is bounded by the model's confidence

The dep_loc hallucination count drops monotonically as the dependentLocalityPenalty is strengthened (defaults 2.0 → 6.0 → ...). The direction is correct. But the absolute reduction is small: even at −6.0 penalty (3× the default), only 19/1063 hallucinations get suppressed. To suppress more, you'd need to push the penalty higher — at which point it starts corrupting legitimate decisions on other tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words..

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s confidence on its synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-induced dep_loc predictions is high. Per the v0.6.1 calibration probe design: when the hallucinations are high-confidence, retraining is required, not just a decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs.-time threshold or bias.

Default morphology bias is over-aggressive and causes collateral street_suffix damage

At default magnitudes (maxAffixBias=3.0), the morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition. inflates street_suffix hallucinations from 204 → 332 — a +63% increase. Investigation: the libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. street_types.txt dictionaries contain many 1-2 character abbreviations (a, b, av, bd, br, ...) that collide with US stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviations (OR, CA, ND, NY) and short tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. A minimum-length-3 surface-form filter mitigates this (av no longer matches, but avenue, rue, blvd still do — see resolver-wof-sqlite/street-morphology-fst-builder.ts's minVariantLength option).

Even with the filter, default bias magnitudes still produce 238 hallucinations vs the 204 baseline — a smaller but real regression. Lowering maxAffixBias to 1.0 brings collateral damage back to 213, basically baseline-equivalent.

Layer 1 is a real deliverable on top of a v0.6.2 retrain

Architecturally, LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 1 IS the dual-FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. integration plumbing that future layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. (LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 1.5 candidacy, LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 2 streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. identity, LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 4 brand FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead.) flow through unchanged. The infrastructure work tonight wasn't speculative — ParseOpts.fstStreetMorphology, buildStreetMorphologyEmissionPriors, the PlacetypeId extension, the two PLACETYPE_ORDER synchronization points, the libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. dictionary walker — all of it is correct and tested. It just needs a backbone modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that wasn't trained to be wrong about dep_loc.

What this means for v0.6.2

Per DeepSeek's turn 2 recipe and the street-supplement architecture doc:

1. Retrain with synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. 0.5 (down from 2.0) — reduces the gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. pressure that pushed the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' into overconfident dep_loc predictions.
2. Explicit O-tags on streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. slots in non-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. rows — the negative-example counter-distribution that synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. is currently missing.
3. LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 1 prior at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. — the morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition. as an additive anchor; meaningful on top of a corrected backbone, insufficient on its own.

The morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition. infrastructure landed in this shift is exactly the inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. plumbing v0.6.2 will use.

Reproducing

# Baseline: v0.6.1 + admin FST only
node scripts/eval-morphology-fst.ts \
  --model $MAILWOMAN_DATA_ROOT/models/quantized/model-v061-step-100000-int8.onnx \
  --tokenizer $MAILWOMAN_DATA_ROOT/models/tokenizer/v0.6.0-a0/tokenizer.model \
  --model-card neural-weights-en-us/model-card.json \
  --admin-fst $MAILWOMAN_DATA_ROOT/wof/fst-per-locale/fst-en-us.bin \
  --golden data/eval/golden/v0.1.2 \
  --no-morphology

# With morphology FST (defaults)
node scripts/eval-morphology-fst.ts \
  --model $MAILWOMAN_DATA_ROOT/models/quantized/model-v061-step-100000-int8.onnx \
  --tokenizer $MAILWOMAN_DATA_ROOT/models/tokenizer/v0.6.0-a0/tokenizer.model \
  --model-card neural-weights-en-us/model-card.json \
  --admin-fst $MAILWOMAN_DATA_ROOT/wof/fst-per-locale/fst-en-us.bin \
  --golden data/eval/golden/v0.1.2

# Tuned (low affix bias, strong dep_loc penalty)
node scripts/eval-morphology-fst.ts \
  --model $MAILWOMAN_DATA_ROOT/models/quantized/model-v061-step-100000-int8.onnx \
  --tokenizer $MAILWOMAN_DATA_ROOT/models/tokenizer/v0.6.0-a0/tokenizer.model \
  --model-card neural-weights-en-us/model-card.json \
  --admin-fst $MAILWOMAN_DATA_ROOT/wof/fst-per-locale/fst-en-us.bin \
  --golden data/eval/golden/v0.1.2 \
  --max-affix-bias 1.0 --max-neighbour-street-bias 1.0 --dep-locality-penalty 6.0

See also

• Street-supplement architecture — the design this evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. validates against
• WOF hierarchy gap — the structural cause
• v0.6.1 error analysis — the original regression
• 2026-05-28 night-2 postmortem — the postmortem that drove the design consult