Coarse-placer M2 — the open-set rule, verdict + decision (#244)

M2 of the coarse-placercoarse-placerA lightweight int8 country classifier (~0.79 MB) that predicts which of a set of target countries an address belongs to, feeding a soft prior into resolver disambiguation. soft countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. prior. Goal (from the soft-signal spec + the OA-breadth verdict): an open-set / novelty method so never-trained off-map families are caught without the in-map cost the raw confidence threshold imposes — to clear the ~88/88 ceiling a linear char-ngram modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' hits, and make a default-on integration defensible.

TL;DR

The fix was a decision rule, not a new modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' or a retrain. The OA-broadened modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' already carries the open-set signal in its OTHER headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.; the ceiling was how we read it. Reading total in-map mass 1 − P(OTHER) as the reject score (and routing on the in-map argmax) decouples "is it in-map?" from "which countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.?" and clears 90/90 post-hoc. Wired as an opt-in openSet rule on CoarsePlacer; the mailwoman geocode --place-country flag uses it.

Phase 1 — post-hoc score comparison (component probe)

On the frozen modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (no retrain), over the leave-one-family-out probe (in-map test 55 k vs off-map HELDOUT 66 k — baltic/oceania/middle-east never trained), honest dev→test (threshold picked on a dev half, frozen on a disjoint test half), metric = min(in-map routing accuracy, heldout-caught):

score	honest dev→test min
p_inmap = 1 − P(OTHER)	91.3 ✅ clears 90/90
maxprob (top in-map softmaxsoftmaxThe function that converts a vector of logits into a probability distribution summing to 1, applied after priors and biases are added to the emission logits. — the old rule)	89.1 (the known ceiling)
maxlogit	83.6
energy = logsumexp(in-map logitslogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities.)	83.4
Mahalanobis (tied-cov, in-map-logitlogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. space)	76.2

Full report: docs/articles/evals/2026-06-14-coarse-placer-m2-openset.md.

Why p_inmap wins (and isn't gaming maxprob): maxprob rejects an address whose probability mass is spread across several in-map countries (e.g. 0.4 FR / 0.4 GB / 0.2 OTHER → top 0.4, rejected) even though it is clearly not off-map. p_inmap sums the in-map mass (0.8 → kept) and lets the argmax route it. So the two jobs — reject vs route — stop fighting over one softmaxsoftmaxThe function that converts a vector of logits into a probability distribution summing to 1, applied after priors and biases are added to the emission logits.. Mahalanobis underperforms because a linear modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s in-map-logitlogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. space isn't a clean Gaussian manifold; the discriminative OTHER headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. is the detector. ⇒ PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 2 (a retrained binary reject-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.) is unnecessary. (Independently sanity-checked with DeepSeek, which had pre-registered Mahalanobis/reject-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.; it agreed the reasoning is sound and the simpler rule is the right call.)

Phase 3 — the assembled-pipeline gate (grade the pipeline, not the component)

Re-ran the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-disambiguation gate (parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. → resolve, the real wiring) on the bundled int8 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' with the open-set rule. Report: docs/articles/evals/2026-06-14-coarse-placer-m2-pipeline-gate.md.

rule	in-map right-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.	wins	regressions	off-map
M1 max-prob (0.9)	64.7 → 85.3 (+20.6pp)	7	0	0→0 graceful
M2 open-set	64.7 → 91.2 (+26.5pp)	9	0	0→0 graceful

The two new wins are exactly the cases the rule targets — Birmingham, AL (US/GB homographhomonymyOne surface, many referents: 'Georgia' is a US state in 'Atlanta, Georgia' and a country in 'Tbilisi, Georgia.' Handled by disambiguation, split across two stages — the parser resolves the tag from in-string context; the resolver late-binds the referent with geographic context.: placer US 0.79, the max-prob rule abstained → resolved to Birmingham UK; open-set keeps it → US) and Los Angeles (US 0.81). Byte-stability invariant (abstain/OTHER ⇒ OFF≡ON) holds; 0 violations.

Decision: the threshold, and the asymmetry

DeepSeek's sharpest point: 90/90 is the wrong objective for a soft prior. The prior never filters — it only re-ranks, tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous.-safe — so a wrong off-map guess costs ~nothing (the M1 + M2 gates both show off-map 0→0, 0 regressions), while a false reject of an in-map address forfeits the disambiguation win. The cost is asymmetric → bias toward in-map recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1., and set the threshold on the assembled pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package., not the component min.

On the assembled gate the threshold is a flat optimum in [0.5, 0.9] (identical 9 wins / 0 regressions — the homographhomonymyOne surface, many referents: 'Georgia' is a US state in 'Atlanta, Georgia' and a country in 'Tbilisi, Georgia.' Handled by disambiguation, split across two stages — the parser resolves the tag from in-string context; the resolver late-binds the referent with geographic context. wins are all high-mass). So we keep abstainBelow 0.9: it captures every win, keeps off-map catch high on the component (≈88% at 0.9-ish), and minimizes in-map misrouting exposure (inject only when confident). The rule — not the threshold — is the M2 lever (+5.9pp in-map over max-prob).

Status + what's gated

• Shipped in M2 (this branch): the openSet rule on CoarsePlacer (opt-in, default-off → byte-stable); --place-country uses it. The whole prior remains default-off overall.
• Misrouting gate — PASSED (the pre-default-on check). The 54-row homographhomonymyOne surface, many referents: 'Georgia' is a US state in 'Atlanta, Georgia' and a country in 'Tbilisi, Georgia.' Handled by disambiguation, split across two stages — the parser resolves the tag from in-string context; the resolver late-binds the referent with geographic context. set under-tests in-map misrouting (the prior injecting a wrong in-map countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.). The across-11 gate (scripts/eval/coarse-placer-inmap-misroute.ts, report docs/articles/evals/2026-06-14-coarse-placer-inmap-misroute.md) resolves 2 000 in-map addresses (200/countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. × 10; TW excluded — 0 WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. rows) with the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. stripped so the countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. must be inferred — the case where the prior can bite. Result: 0 misroutes, 0 regressions, 58 wins, right-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. 50.7 → 53.6%. The tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous.-safe soft re-rank never pushed an in-map address to a wrong in-map countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., even on OOD-parsed inputs. (Caveat — honest read: absolute rates are depressed by the en-US modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' being OOD on non-US addresses + thin WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. for NL/DE/KR, so the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. is conservative; it cleanly answers the misroute question but can't fully validate resolution quality on thin-coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. Production flows usually also pin localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for./defaultCountry.) ⇒ default-on is defensible; the flip is the operator's call (a user-facing default change).
• Residual upgrade — SHIPPED (#244): the soft priorsoft priorOutside knowledge fed to the model or resolver as overridable evidence (a feature, a score term) rather than a hard filter. A focus-country hint becomes an anchor feature; a focus-point becomes a ranking term. now feeds the full in-map posteriorposterior (country posterior)A per-country probability map — e.g. {GB: 0.8, FR: 0.06} — expressing how likely an input belongs to each country, produced by a model (the coarse placer's softmax, or a postcode anchor's lookup) AFTER seeing the input (hence 'posterior', from Bayesian usage: the updated belief). The resolver consumes it as a soft ranking boost (anchorPosterior): each candidate's score gains weight × posterior[candidate.country]. Values lie in [0, 1] and need not sum to 1 (in-map marginals exclude the OTHER class). distribution as the anchorPosterior (vs the one-hot argmax) — the placer hands the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. every plausible countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. weighted by mass, so the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. breaks countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-ambiguous ties with its own place-level evidence and can never be committed to a wrong argmax. New inMapPosterior() on the core placer; placeCountry stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). gained an optional posterior (consumers fall back to the one-hot when absent); the default placer emits it. A/B on both gates (--distribution): homographhomonymyOne surface, many referents: 'Georgia' is a US state in 'Atlanta, Georgia' and a country in 'Tbilisi, Georgia.' Handled by disambiguation, split across two stages — the parser resolves the tag from in-string context; the resolver late-binds the referent with geographic context. identical (91.2%, 9 wins, 0 regressions — those wins are confidently single-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., so distribution ≈ one-hot), misroute +1 win (59 vs 58), still 0 regressions / 0 misroutes — a strict, principled improvement (larger on data with more in-map ambiguity, e.g. European cross-border namesakes).

Reproduce

yarn compile
# Phase 1 — post-hoc score comparison on the frozen model:
node scripts/coarse-placer/eval-openset.mjs --model $MAILWOMAN_DATA_ROOT/coarse-placer/model \
  --out-md docs/articles/evals/2026-06-14-coarse-placer-m2-openset.md
# Phase 3 — assembled-pipeline gate with the open-set rule (bundled int8 model):
node scripts/eval/coarse-placer-country-disambig.ts --openset --abstain-below 0.9 \
  --out-md docs/articles/evals/2026-06-14-coarse-placer-m2-pipeline-gate.md