Skip to main content

Route A Phase I — joint-decode vs argmax baseline (2026-06-07)

The decision document for epic #421. Before committing to making the joint-decode / concordanceconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement. stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). the default, we measured the existing opt-in path (forceJointReconcile) headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. against the argmax default. The result is decisive, and it points the opposite way from what you'd hope.

Verdict: STAY — do not flip the default. Phase II (phrase grouper) is a hard gate, not an option.​

Joint-decode wins enormously on the case it was built for and catastrophically regresses everything else, for one diagnosable reason. The "just flip the default" option is dead; the full programme is the only viable path, and it's blocked on a prerequisite.

What we measured​

scripts/eval/joint-vs-argmax.ts runs the same runtime pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. twice per address — default vs forceJointReconcile — over the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. samples (v0.9.4 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'). Latency is warmed and the run-order is alternated per row so neither path eats the ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. cold-start.

localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.argmax locjoint locΔ locregressedimprovedlatency p99 ×
DE international (city-statedual-role placeA place that is two placetypes at once — Berlin as both a city and a state, Washington DC as city and district. Resolved using the coincident-roles relation plus hierarchy completion. collision — the target)72.2%97.2%+25.0pp2.4%26.8%1.45
US (native)98.8%97.4%−1.4pp2.2%0.8%1.75
FR (native)97.5%97.8%+0.3pp2.0%2.3%2.07
NL (native)99.5%84.0%−15.5pp16.0%0.3%1.63
IT (native)84.8%68.5%−16.3pp26.0%9.8%1.73
ES (native)83.8%58.5%−25.3pp34.0%8.8%1.62

The gate (from #421) wants a regression rate ≤ 0.5% to even consider flipping the default. The multi-word localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. come in at 16–34% — fifty to seventy times the bar.

Why — the phrase grouper, exactly as predicted​

The win and the lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. have the same root. The reconciler decodes over the spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?' proposes; when those proposals don't cover a multi-word component, it falls back to single-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. That's a featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. for the city-statedual-role placeA place that is two placetypes at once — Berlin as both a city and a state, Washington DC as city and district. Resolved using the coincident-roles relation plus hierarchy completion. collision (…, Berlin, Berlin 10115 — the second Berlin is a single tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. the reconciler can re-tag as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., which argmax drops — hence +25pp). It's a disaster for native-order localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. whose place namestoponymA proper name for a geographic place. are multi-word:

"Reggio nell'Emilia"          argmax → "Reggio nell'Emilia" ✓     joint → "Reggio"        (fragmented)
"Las Palmas de Gran Canaria"  argmax → "de Gran Canaria"          joint → "CALLE MAYOR"   (grabbed the street)

Italian, Spanish, and Dutch are dense with multi-word localities, so they take the full brunt; US and French, mostly single-word, barely move. The improvement column confirms the upside is real (DE 26.8%, IT 9.8%, ES 8.8% of rows get better) — joint decodingjoint decodingA decode strategy where the neural model's proposals and rule-based solver's output are reconciled into a single parse tree. Formerly the default (Route A Phase II), now retired in favor of argmax after it was found to break the street+house_number geocode precondition. recovers cases argmax can't — but it's swamped by the fragmentation it introduces.

Latency is a non-issue: p50 is unchanged across the board and p99 sits at 1.5–2.1×, comfortably under the 3.0× ceiling. The blocker is purely quality.

What this means for the plan​

  • JUST-FLIP is dead. Flipping forceJointReconcile to default today would tank localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. accuracy on three of six localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. by 15–25 points. No telemetry gate survives that.
  • The dual-role / multi-role work was the right call. The city-statedual-role placeA place that is two placetypes at once — Berlin as both a city and a state, Washington DC as city and district. Resolved using the coincident-roles relation plus hierarchy completion. recovery it ships (Berlin → 80.9% PIP) is exactly the +25pp that joint-decode also delivers — but the Resolve-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). relation completion delivers it without the multi-word collateral. That mechanism stays.
  • FULL Route A hinges entirely on PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). II (#425) — maturing the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?' to propose multi-word spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. reliably. Until the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?' covers Reggio nell'Emilia as one spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree., joint decodingjoint decodingA decode strategy where the neural model's proposals and rule-based solver's output are reconciled into a single parse tree. Formerly the default (Route A Phase II), now retired in favor of argmax after it was found to break the street+house_number geocode precondition. cannot be the default. This A/B is the re-gate: after #425, re-run it; flip only if the native-order regression drops under the bar.

So PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). I did its job — it cost a day and a benchmark, and it turned "should we spend weeks making concordanceconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement. the default?" into "not until the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?' can cover multi-word spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree., and here's the test that proves when it can." PhasesphaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). III–IV (concordanceconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement. dual-role signal, flip) stay closed behind that.

Harness: scripts/eval/joint-vs-argmax.ts. Numbers generated; per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. JSON under the run's --out-json.