Postcode-only diagnostic (v0.6.0)

Date: 2026-05-29 Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': model-v060-step-100000-int8.onnx TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.: v0.6.0-a0 Sources: mailwoman/test/*.test.ts + data/eval/falsehoods/*.jsonl + data/eval/golden/v0.1.2/*.jsonl Sample size: 3,096 postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-bearing entries, 14 countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. buckets

Built in response to DeepSeek's turn-12 question: "Are we actually bad at postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. per countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.?" PostcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. patterns (CEDEXCEDEX (Courrier d'Entreprise à Distribution Exceptionnelle). A French postal routing for high-volume business mail: a CEDEX code delivers directly from a sorting centre, bypassing the local post office. A common negative-space format Mailwoman must parse. = FR, A1A 1A1 = CA, SW1A 1AA = GB, 12345-6789 = US) are regex-detectable shapes; the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' SHOULD crush this. Earlier v0.6.x evalsevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. showed postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. in the 70-80% range — well below the 95%+ that pattern-matching would imply. This diagnostic isolates postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-only accuracy from the rest of the parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package..

Headline result

Overall: 75.9% (2349/3096) exact-match accuracy.

CountrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.	Total	Match	Rate
US	1,765	1,421	80.5%
FR	1,266	888	70.1%
AU	10	9	90.0%
DE	7	4	57.1%
GB	6	0	0.0%
CA	5	0	0.0%
NL	4	0	0.0%
PT	3	1	33.3%
BR	1	0	0.0%
ES	1	0	0.0%
PL	1	0	0.0%
JP	1	0	0.0%
IN	1	1	100%
UNKNOWN	25	25	100%

The UNKNOWN bucket is from tests that don't carry an explicit localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. (intersection.test.ts, etc., all US-context). The 0% buckets are small samples but consistent: every multi-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. alphanumeric postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. pattern (GB, CA, NL) fails uniformly. US scores 80.5% on a sample of 1,765 — large enough to be reliable.

The failure mechanism is visible in the data

Three distinct patterns of fragmentation:

1. Total miss (alphanumeric, multi-token postcodes)

London SW1A 1AA              → expected "SW1A 1AA", actual (missing)
Edinburgh EH8 9YL            → expected "EH8 9YL", actual (missing)
Piccadilly, London, W1J 9PN  → expected "W1J 9PN", actual (missing)

GB postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. (and similar shapes) are completely skipped. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' either emits no postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. at all or labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. these tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. as something else (often localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. or O).

2. Truncation (postcode label attaches to ONE token of a multi-token postcode)

Toronto, ON M5V 2T6, Canada           → "M5V 2T6" → actual "2T6"
8 Seven Gardens Burgh, ..., IP13 6SU  → "IP13 6SU" → actual "P13"
6 Elm Avenue, ..., Birmingham, B12 8QX → "B12 8QX" → actual "B12"
H2X 2T6                               → actual "2"
H3B 1A3                               → actual "H3B"
1012 AB Amsterdam                     → "1012 AB" → actual "1012"
東京都中央区銀座 100-0001              → "100-0001" → actual "0001"

Either the alphabetic chunk or the numeric chunk gets labeled — never both. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' treats them as separate spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree..

3. Character-level drift on numeric postcodes

Am Bürgerpark 15-18, 13156, Berlin → "13156" → actual "3156"
4 Cité Du Cardinal Lemoine 75005 Paris → "75005" → actual "5005 Paris"
1 bis Av. Amélie, 92320 Châtillon  → "92320" → actual "2320"
Żorska 11, 47-400                  → "47-400" → actual "7-400"
Send mail to: 200 Elm St, Springfield, IL 62701, USA, Earth → "62701" → "2701"

Numeric postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. drop a leading digit. The pattern 5005 Paris suggests the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. boundary smeared right into the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy..

The mechanism: tokenizer fragmentation

SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. with a 48K vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. tokenizes US-dominant postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. patterns fairly cleanly (62701 → [62, ##701] or [6, ##27, ##01]), but alphanumeric patterns get split aggressively:

"SW1A 1AA" → ["S", "##W", "##1", "##A", "▁", "1", "##AA"]
"M5V 2T6"  → ["M", "##5", "##V", "▁", "2", "##T", "##6"]
"1012 AB"  → ["1", "##0", "##1", "##2", "▁", "AB"]

The seven-character pattern that regex would crush is invisible to a subword modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' has to learn the relationship between disconnected subword fragments by purely statistical co-occurrence. With GB postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. overwhelmingly absent from trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., that statistical signal never gets strong enough to consistently spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. the whole pattern.

The numeric drift (62701 → 2701) is a different but related mechanism. SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. may tokenize 62701 as [6, ##27, ##01] or [62, ##70, ##1] depending on context; the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. SOME of these tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. as postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. but not all. If the first tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. (6 or 62) ends up attached to a different spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. (here, a preceding ,), the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. starts at the second tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. and we lose the leading digit.

How this changes the v0.7 plan

This diagnostic was DeepSeek turn 12's "highest-information secondary experiment." The decision tree was:

Calibration	PostcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.	Action
Improved + overconfidence dropped	< 90% on some countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.	v0.7 = calibration + postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. fix
Improved + overconfidence dropped	≥ 90% everywhere	v0.7 = calibration only
Flat + overconfidence unchanged	< 90%	v0.7 = structural pivot
Flat + overconfidence unchanged	≥ 90% everywhere	Investigate further

The diagnostic settles the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. column decisively: postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. are

< 90% in MOST countries with samples (US 80.5%, FR 70.1%, GB/CA/NL 0%, DE 57%, PT 33%). The postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. fix is on v0.7's path regardless of how calibration shakes out.

Candidate fixes

1. Character-level featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. extractor. Add a parallel character-CNN or character-bigram embeddingembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding. layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. that processes the raw character stream alongside the subword tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. The postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. shape becomes visible at the character level even when fragmented at the subword level.
2. PostcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-aware tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. pre-pass. A regex pass that detects-and-protects postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-shaped substrings (/[A-Z]\d[A-Z]?\s*\d[A-Z]\d/, etc.) before tokenization. Single tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., single labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., no fragmentation.
3. Explicit per-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead.. Treat postcodespostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. the same way we treat admin names — a deterministic FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. that recognizes countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.- specific shapes and biases the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' toward labeling the matched spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. as postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.. LayerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. 1.5 of the shallow-fusion architecture.
4. Train on more diverse postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. data. OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries.-EU covers GB, CA, NL, DE; ingesting it would give the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' real-world distribution of the patterns it currently misses. Doesn't address tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. fragmentation but reduces the data-side amplifier.

The DeepSeek turn-12 recommendation favors options 1 + 3 (character featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. + postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead.) for v0.7 if calibration alone doesn't close the gap.

Reproducing this diagnostic

node scripts/harness-postcode.ts \
  --model $MAILWOMAN_DATA_ROOT/models/quantized/model-v060-step-100000-int8.onnx \
  --tokenizer $MAILWOMAN_DATA_ROOT/models/tokenizer/v0.6.0-a0/tokenizer.model \
  --model-card neural-weights-en-us/model-card.json \
  --tests mailwoman/test \
  --falsehoods data/eval/falsehoods \
  --golden data/eval/golden/v0.1.2 \
  --out-json /tmp/postcode-harness-v060.json

For gate use in CI:

node scripts/harness-postcode.ts ... --gate --floor 0.9 --min-count 10

Exits nonzero on any countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. with ≥ 10 samples and accuracy below the floor. v0.6.0's US (80.5%, 1765 samples) fails this gate today; v0.7's target is for it to pass.

See also

• v0.6.x cycle retrospective + v0.7 plan
• FST priors as shallow fusion — the architectural pattern that a postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. would slot into
• scripts/harness-postcode.ts — the harness itself