T1a — the regression cross-tab: the falsifier fired, and the metric can't see why
Question (pre-registered, 2026-07-16 review TiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. 1a): the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode netsneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network. ~0 overall but +23.8pp on Paris, so it is losing somewhere. Name the class before shipping. Pre-registered kill condition: "if the regression class is 'streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. hallucinated where none exists', that's a NEW failure mode and the flag stays off-by-default."
Verdict: the kill condition is met. The flag stays OFF by default.
The spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode invents streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. on inputs that have none — a failure class the shipped decode does not have to this degree, and one that every streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. metric in the arc is structurally incapable of measuring.
This doc first said the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode hallucinates "58% more often" (12/54 → 19/54). That framing
overstatesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. what n=54 can support: the two intervals overlap, and McNemar on the paired discordants
(11 vs 4) gives p = 0.12. The rate difference is not established. What IS established is that
the 11 new hallucinations are real, individually inspectable failures — New York, NY → streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=new york is not a statistical claim — and the pre-registered condition turns on the failure CLASS
existing, not on its rate being significantly worse. The verdict stands on those grounds. The rate
question is settled by the T1c fragment board, whose bare-locality class carries n=400 (±4.0pp).
Caught by the same discipline that produced the finding; left visible rather than quietly edited.
| Probe | scratchpad/t1a-regression-crosstab.mjs |
| Modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' | v264 (shipped, md5 3e534072) vs v301 (spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads., md5 add5b344) |
| Config | production (query-shape prior fed — ffcb8e96) |
| Fixture | parity-corpus.triaged.jsonl — 321 live, 267 streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-gold, 54 streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-free |
1. The tie is not a tie — it's 17-for-17 churn
| v301 seg ✓ | v301 seg ✗ | |
|---|---|---|
| v264 tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. ✓ | 137 | 17 ← regression |
| v264 tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. ✗ | 17 | 96 |
Netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network. +0 on streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. rank-1 (matching the aggregate: 154/267 both ways). But the decode changes 34 fixtures to get there. An aggregate tie was hiding a full third of the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. moving.
The within-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' cross-tab (v301 tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. × v301 seg) is the same shape: +1 netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network., 18 fixed / 17 broke.
2. The win class is exactly the thesis
All 17 are the target class — the tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. decode is too timid and the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode recovers the phrase:
FR "Rue de Paris" v264="" seg="rue de paris" ✓
FR "Allée Victor Hugo" v264="" seg="allée victor hugo" ✓
FR "Esplanade Méditerranée" v264="" seg="esplanade méditerranée" ✓
ZZ "Foostraße" v264="" seg="foostraße" ✓
FR "Place Sohier Vervins" v264="" seg="place sohier" ✓
FR "Avenue Aristide Briand" v264="aristide briand" seg=+prefix ✓
NO "Maria Dehlis vei 15" v264="vei" seg="maria dehlis vei" ✓
SE "Gamla Varmdovägen 6" v264="varmdovägen" seg="gamla varmdovägen" ✓
Five of them are cases where the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emits nothing at all. The spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode is doing the job it was built for, on the class it was built for.
3. The regression class is systematic — 8 of 17 are digit-eating
CZ "Korunní 810, Praha" gold="korunní" seg="korunní 8"
ES "Carrer d'Aragó 155 08011" gold="carrer d'aragó" seg="carrer d'aragó 1"
FR "4 Cité Du Cardinal Lemoine 75005" gold="cité du cardinal lemoine" seg="… lemoine 7"
NO "Øvste Skogen 121" gold="øvste skogen" seg="øvste skogen 1"
PL "Ulica Strzelecka 12" gold="ulica strzelecka" seg="ulica strzelecka 1"
PL "Żorska 11, 47-400" gold="żorska" seg="żorska 1"
SK "Divadelná 41/3, Trnava" gold="divadelná" seg="divadelná 4"
SE "Ångermannagatan 80, Vällingby" gold="ångermannagatan" seg="ångermannagatan 8"
The streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. swallows the first digit of the adjacent house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. or postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. — never the whole
number, always the first piece. The tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. splits numbers into digit piecesECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). (▁8|1|0), and the
segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context. ends one piece late. Eight localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for., one shape. This is not noise; it is a boundary off-by-one at
a digit run, and it looks fixable independently of everything else here.
Three more are truncationstruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. ("bulevardul iuliu maniu" → "bulevardul i", cutting mid-word), which is
the same boundary error in the other direction.
4. The falsifier: hallucination on the 54 rows nothing measures
Every streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. harness in the arc filters to fixtures with expect.street — seg@1, oracle@5,
oracle@10, the Paris board, all of them. So a spurious streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. on a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-only row is invisible by
construction. 321 live fixtures, 267 streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-gold ⇒ 54 rows where this hides.
| decode | emits a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. on a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-free row | 95% Wilson |
|---|---|---|
| v264 tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 (shipped) | 12/54 = 0.222 | [0.132, 0.349] |
| v301 tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 | 12/54 = 0.222 | [0.132, 0.349] |
| v301 seg@1 (the flag) | 19/54 = 0.352 | [0.238, 0.485] |
11 new hallucinations introduced, 4 removed. Netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network. +7. They are not marginal:
US "New York, NY" → street="new york"
US "New York, New York" → street="new york"
NL "Rozenburg" → street="rozenburg"
ZZ "new south wales aus" → street="new south wales"
AU "BOOM" → street="boom"
US "philadelphia museum of art" → street="philadelphia museum of art"
Six are pure localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. rows (indefensible). Five are venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. rows (softer — a venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. read as a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. is arguable). Both partitions get worse: pure +6/−3, venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. +5/−1.
Not a grammar defect. The exported grammar carries explicit start_transitions, end_transitions,
and an O segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context. type at index 0 — an all-O parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. is representable and cheap to express. The decode
chooses streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. anyway. This is learned miscalibration, not a structural hole. (Checked, because the
structural story was the attractive one.)
4.1 What n=54 can and cannot support
The intervals above overlap. Treating the two rates as independent samples, 0.222 vs 0.352 is not a distinguishable difference at this n. The right test is paired — same 54 rows, two decodes — so McNemar on the discordants:
b = 11 (v264 silent, seg hallucinates) exact two-sided p = 0.1185
c = 4 (v264 hallucinates, seg silent) => direction consistent, NOT significant at 0.05
So: the rate difference is not established. Anyone quoting "+58%" from this page is quoting noise with a decimal point — the same sin the fragment board exists to stop, committed here first.
Three things survive that arithmetic, and they are what the verdict rests on:
- The 11 failures are real, not inferred.
New York, NY→ streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=new yorkis a defect you can read, reproduce, and fix. Its existence is not a statistical claim and no p-value bears on it. - The pre-registered condition turns on the class, not the rate. It reads: "if the regression class is 'streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. hallucinated where none exists', that's a NEW failure mode and the flag stays off-by-default." The class exists. The condition fires as written.
- Precaution is asymmetric here. The flag is opt-in either way; the cost of holding it is a consumer types a flag, and the cost of shipping it wrong is silent streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. hallucination in a geocoder. At p=0.12 you do not get to round toward the convenient answer.
The rate question is answered by T1c, not by this page. The fragment board's bare-locality class
runs n=400 — a ±4.0pp interval at p≈0.22, versus ±10.9pp here. That is the measurement; this was the
detection.
5. One property, three consequences
The win class, the digit-eating, and the hallucination are not three findings. They are one:
The spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode is over-eager to emit and extend a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context..
- Where a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. exists and the tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. decode was too timid → it wins (+17).
- Where the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. boundary abuts a digit run → it over-extends (+8 of the 17 regressions).
- Where no streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. exists at all → it invents one (+11 hallucinations).
Netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network. on the rows the metric scores: +0. Netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network. on the rows the metric drops: −7. Counting both, the flag is netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network. negative — and the arc has never once measured it that way.
6. Consequences
For the ship decision (#42): the flag stays OFF by default. The pre-registered kill condition fired on its exact terms. The decode's value on the target class is real and large — five fixtures where the shipped modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' says nothing — but it is bought with a hallucination rate the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. metric cannot price. A consumer that wants the k-best list can opt in; nothing gets it by default until §5's over-eagerness is separated from §2's win.
For the instrument: the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. metric's expect.street filter is a blind spot that flatters the
spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode specifically, because the decode's failure mode lives exactly in the rows the filter drops.
This is the third instrument failure in one arc (PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1's starvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). channels, PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 4a's dark resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.,
now this), and the pattern is the same every time: the harness could not see the thing that was wrong.
baselines.json cannot catch it either — a baseline over a filtered corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. is self-consistent forever.
The two-board structure (#43) needs a third question: what does the board not score?
For T2 (the BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.) — this is the strongest evidence yet. The hallucination and the bare-fragment failure are the same miscalibration seen from opposite sides. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. is thin on bare streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. fragments and thick on bare localities; the BIO headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. learned "bare toponymtoponymA proper name for a geographic place. → localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy." (the §2 prior that starts the arc), and the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads., trained on the same corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., learned an over-eager streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context. that fires on bare toponymstoponymA proper name for a geographic place. regardless of whether one is there. Both headsattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. are miscalibrated on the same axis, in opposite directions, from the same missing data. A shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. carrying real bare streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. and real bare localities is the one intervention that speaks to both.
Reproduce: node scratchpad/t1a-regression-crosstab.mjs (writes per-fixture records to
scratchpad/t1a-crosstab.json).