Skip to main content

The span decode's ship case is refuted — by the shard, on the span decode's own target class

Question: with T2 landed, would we still ship the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode behind a flag, as §6 of the review follow-up unanimously recommended?

Answer: no — and not for the reason we pre-registered. The flag rested on one number, and a plain tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. decode on the shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.-trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' beats that number by 8 fixtures.

1. The measurement​

Paris target class, n=63, production config, int8 against int8:

95% Wilson
v264 token@1 (shipped)33/63 = 0.524[0.403, 0.642]
v301 seg@1 — the flag's entire case48/63 = 0.762[0.644, 0.850]
v310 token@1 — the shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., no spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.56/63 = 0.889[0.788, 0.945]
the flag's claim : v264 token -> v301 seg   = +23.8pp
the shard alone  : v264 token -> v310 token = +36.5pp
span decode's MARGINAL value over the shard : -12.7pp

§6 valued the flag "for the list and the target class (+23.8pp Paris)". On that target class the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode is now second best, behind a simpler thing. An opt-in path that loses to the default is not a featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them..

2. What this does and does not establish​

It does not establish "spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode < tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. decode." The comparison is between artifacts — v301 (old corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.) + spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. vs v310 (new corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.) + tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. Two variables move at once.

It is exactly the ship question. "Should we ship this flag?" is a question about artifacts, and the answer is no: a strictly better and strictly simpler artifact exists. Nobody would opt into the worse one.

3. Why §6 was right and is now wrong​

§6 recommended the flag on the belief that the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode owned the bare-fragment class — that the +23.8pp was the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.'s to give. T1c and T2 falsified the belief:

  • T1c found the cause: the house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. is a licence, not a hint. Bare streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. read as localities because the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. distribution taught that, not because the decode couldn't express the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree..
  • T2 fixed the distribution. The class the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. decode was ordered to rescue got rescued by data — further, and without a decode change.

The decision was correct given what was known. The premise moved underneath it. That is what a pre-registered plan is for: §6's own text made the flag conditional ("TiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. 1a decides whether the flag has a regression cost we haven't named yet"), and the tierstierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. below it kept going.

4. Off-by-default is not free​

The tempting move is "ship it off by default, it costs nothing." It costs:

  • an API surface we have to keep;
  • a maintenance burden on a decode path nobody is validating;
  • an implicit endorsement — shipping weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. trained on a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. we have since replaced;
  • and a worse experience for anyone who opts in, which is the only population it has.

5. The one open question — and it is the last one​

Everything above compares artifacts. The within-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' question has never been asked on a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. that teaches bare streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., because no spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. has ever been trained on one. That is v3.2.0-fragment-span (run ap-jcRsH8TNQny84vVeOWLG8f), one variable off v310: the headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads..

Pre-registered kill shot: seg@1 must beat v310's token@1 = 56/63 (0.889) — the bar the flag failed by 12.7pp. If a shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.-trained spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. cannot beat a shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.-trained plain decode on the target class, the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. has no case at any weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., and the arc closes. No span_loss_weight tuning, no re-run: that is the treadmill the arc's own guard forbids.

6. What the arc bought anyway​

Four phasesphaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary)., and — as of this page — no shippable artifact. That is worth saying plainly rather than dressing up.

What it produced instead is the reason the fix exists at all. oracle@k and the k-best decode were the instruments that made the headroom visible (oracle@10 0.775 against a 0.577 rank-1 — a gap nobody could see when every gate scored top-1). That gap is what motivated T1a's cross-tab, which found the digit-eating and forced the hallucination check, which motivated T1c's board, which found the licence, which built T2's shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., which fixed the class.

The arc asked "can a better decode find the right answer?" The reply turned out to be: the right answer was never in the list to be found — it wasn't in the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data. The instrument that proved the answer wasn't there is what led us to put it there.

Reproduce: node scratchpad/paris-3way.mjs.