Demo Preset Diagnosis — v0.5.1 Locality/Region Confusion

Fixes shipped

• "New York, NY": Fixed by the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-aware localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. bias guard (#174). The QueryShape prior now skips localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. bias when the preceding text matches the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.'s full name.
• FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. importance weighting: Washington DC localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. (0.815) correctly outranks Washington stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. (0.764) via Wikipedia importanceWikipedia importanceA place-notability score derived from the count and language spread of a place's Wikipedia articles, normalized to [0, 1]. A resolver ranking prior that helps pick the prominent same-named place. (#173).
• "Washington, DC": Partially improved — FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. biases toward localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., but the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s high B-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. confidence after streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. phrases resists the prior. Remaining fix is in trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data quality.
• "San Francisco": Fixed in v0.5.2 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — correctly labeled as localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy..

Documents the root-cause analysis of v0.5.1's demo preset failures and the fix plan. Derived from an 8-turn DeepSeek consultation (2026-05-25).

The failures

Six demo presets tested against v0.5.1 (h384, val_macro_f1=0.638). Two systematic failures:

Preset	Input	Expected	Got
#1	1600 Pennsylvania Avenue NW, Washington, DC 20500	Washington=localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., DC=regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.	Washington=regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., DC=regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.
#2	350 5th Ave, New York, NY 10118	New York=localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., NY=regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.	New York=regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., NY=regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.

Other failures (#3 venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label., #4 directional, #6 postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.) are unrelated — covered separately below.

Root cause: WOF bare-name frequency dominance

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' receives no positional signal about where a tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. appears in an address structure. v0's rule-based solvers had explicit positional penalties (HouseNumberPositionPenalty, PostcodePositionPenalty) that the neural pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. does not replicate.

The transformerneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' backbone HAS positional encodings and CAN learn "tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. after streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., before regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation → localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.." But:

• WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. entries are bare place namestoponymA proper name for a geographic place. — "Washington" → B-region with NO surrounding context
• OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. entries are full addresses — "Washington" → B-locality between a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. and a regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation
• WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. bare-name entries outnumber OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. positional-context entries for ambiguous place namestoponymA proper name for a geographic place.

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learns the frequency-dominant pattern (Washington = regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.) because it sees that labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. more often in isolation. The positional signal from OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. full-address rows exists but is diluted.

BIO boundaries are correct

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emits B-region for "Washington" and a separate B-region for "DC" (not I-region). This is structurally correct — they ARE separate entities separated by comma+space (O-tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.). The CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. transition mask is working. The problem is tag assignment on the first entity, not boundary detection.

Fix plan (priority order)

1. QueryShape locality soft prior (inference-time, no retraining)

Detect unambiguous 2-letter regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation (e.g., DC, NY, CA) → bias preceding place-name tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. toward B-locality / I-locality via emission prioremission priorA log-probability bias injected into the model's emission logits from an external signal — gazetteer frequency, an FST, a Wikipedia-importance score — combined with the learned weights at decode time..

• Where: Between classifier forward pass and CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores. decode (existing buildEmissionPriors path)
• Magnitude: +2.0 logitlogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. boost (same as existing format-hit priors; ~7.4× odds multiplier at softmaxsoftmaxThe function that converts a vector of logits into a probability distribution summing to 1, applied after priors and biases are added to the emission logits.)
• Safety constraint: Only bias tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. verified against WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. entries at the detected localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. "Pennsylvania" preceding DC stays as streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. because it's NOT a WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. in the relevant context.
• Effort: ~20 lines. Zero retraining.
• Test: Demo presets #1 and #2. If still fails, bump to +3.0. If overcorrects on "Washington stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality." inputs, pull back.

2. Source weight rebalance (training-time)

source_weights:
  osm: 1.0
  wof: 0.3

Reduces WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. bare-name frequency dominance. OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. positional signal becomes the primary teacher for ambiguous place namestoponymA proper name for a geographic place.. One config line.

3. Training augmentation — directional + region-abbreviation expansion

Two independent augmentations:

• Directional: Train on both "350 5th Ave NW" and "350 5th Ave Northwest" with identical BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components.
• RegionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation: Train on both "NY" → B-region and "New York" → B-region I-region

Both scoped to unambiguous expansions only. ~60 lines total in corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package..

Do NOT normalizenormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input. at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. time. SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. was learned from raw corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. distribution; feeding normalized forms at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. creates a train/test distribution mismatch. NormalizenormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input. at trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time only.

4. Reconciler as default path (architecture)

• Graceful degradation: no WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. data = ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores.-only (current behavior). Emit { reconciled: false, reason: "wof-missing" } on result.
• Optional @mailwoman/reconciler-data-en-us package (~5 MB): localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. rows { id, name, parent_id } + regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. rows { id, name }. Enables concordanceconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement. scoring for npm users.
• Full WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. (~2 GB) remains the high-accuracy path for self-hosted deployments.

Other preset failures (different root causes)

#	Preset	Failure	Root cause	Fix
3	Pier 39	venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.=0%	No venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. annotations at corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. scale	Future: venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. Now: hybrid mode catches via rule chain
4	W Addison St	directional prefix missed	Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' hasn't seen enough directional abbreviations	Directional augmentation (#3 above) + hybrid rule fallback
5	(passes)	—	—	—
6	90210	postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. confidence low	Neural postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. F1 low (1.6%)	Rules handle (98.8% precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. in hybrid mode)

Key insight: switch demo to hybrid mode

If the demo currently runs neural-only, switching to hybrid mode immediately fixes presets #3, #4, and #6 via rule fallback — zero code changes beyond the mode flag.

Pre-tokenization normalization — decision: NO

Normalization	At inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.?	At trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.?
Directionals (NW → Northwest)	No — subword mismatch	Yes — unambiguous, high-frequency
St → StreetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./Saint	No — requires disambiguation	No — modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learns from context
Punctuation	SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. handles it	No
UnitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. notation (#)	No — too many edge cases	No

Relationship to QueryShape

The localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. soft priorsoft priorOutside knowledge fed to the model or resolver as overridable evidence (a feature, a score term) rather than a hard filter. A focus-country hint becomes an anchor feature; a focus-point becomes a ranking term. extends the existing QueryShape emission-prior system. New field on the QueryShape result:

regionAbbreviations?: Array<{ start: number; span: string }>

Detected at StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2.5 alongside format-hit scorers (regex: /,\s*[A-Z]{2}\b/ for en-us). In buildEmissionPriors, for each detected abbreviation, bias preceding WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-verified localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. toward B-locality / I-locality.

This fits the bitter-lesson framing test: the regex is localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.-bounded (one pattern per localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.'s abbreviation convention), not a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. lookup. The WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. verification is a safety rail, not the primary signal.

See also

• QUERY_SHAPE.md — the sub-system this extends
• TRAINING_RECIPE_LEVERS.md — trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. knobs including source_weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.
• ../../concepts/dual-loss-curvature-conflict.md — why CE-only trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.