Parity Scorecard — 2026-06-10 (baseline: v4.2.0, the consolidation flag-plant)
Supersedes 2026-06-09. Same two lenses, same rules:
arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. is whole-parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.-strict (honest, understatesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. per-tag wins); per-tag F1 is
what the campaign moves; real-OOD columns are the truth for campaign tags. Self-emitted
from external-arenas.sh + per-locale-f1.ts + the real-OOD scorers — do not hand-edit.
What changed since 06-09: the v1.0 consolidation campaign concluded (Runs A/B/C, the
measured 29M stability ceiling — see 2026-06-10-consolidation-session.md), Run B shipped
as v4.2.0 after a 4/4 ship gate (2026-06-10-night-10-ship-gate.md).
Lens 1 — capability arenas (v4.2.0 int8, TRUE ship config: anchor + gazetteer fed)
| arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. | n | v0 | v4.1.0¹ | v4.2.0 | both | neural-only | v0-only | both-fail |
|---|---|---|---|---|---|---|---|---|
| libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. (clean/canonical) | 69 | 29% | 22% | 41% | 20% | 20% | 9% | 51% |
| perturb (noisy/degraded) | 398 | 39% | 62% | 71% | 35% | 36% | 5% | 25% |
| postal (edge formats) | 38 | 26% | 11% | 18% | 11% | 8% | 16% | 66% |
¹ v4.1.0 anchor-fed (its true config; the gaz channel doesn't exist for it).
MEASUREMENT CORRECTION (night-10): every prior arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. number for an anchor-trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (v4.0.0 onward) was graded with ZERO-FILLED anchor/gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. channels — the harness couldn't feed them until tonight (
--gazetteer-lexicon/--anchor-lookup). The first-pass "v4.2.0 dip" (19/58/8) was entirely this artifact. In ship configuration, v4.2.0 leads v0 on the CLEAN arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. for the first time in project history (41 vs 29) while extending the noisy lead to +32pp. The routing truth is now: neural leads clean AND noisy; v0 leads only rare edge formats (po-box/military/rural — #492's riders). The remainingv0-onlycells (9/5/16) are #478's bounty.
Lens 2 — per-tag F1 (int8, gaz-fed, golden dev + real-OOD)
| Tag | US (4.1.0 → 4.2.0) | FR (4.1.0 → 4.2.0) | status |
|---|---|---|---|
| postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. | 98.3 → 97.3 | 99.5 → 99.6 | ✅ (stated −1.0 US) |
| house_number | 96.2 → 96.9 | 91.2 → 94.6 | ✅ FR best ever |
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. | 60.1 → 72.9 | 69.7 → 70.7 | ✅ +12.8 |
| regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. | 78.4 → 89.1 | 27.8 → 27.6 | ✅ US / ❌ FR (#330) |
| countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. (homographhomonymyOne surface, many referents: 'Georgia' is a US state in 'Atlanta, Georgia' and a country in 'Tbilisi, Georgia.' Handled by disambiguation, split across two stages — the parser resolves the tag from in-string context; the resolver late-binds the referent with geographic context.) | ~0 → 89.8 | — | ✅ the lever, banked |
| streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. (folded) | 78.5 → 76.2 | 60.1 → 58.2 | ◐ stated re-baseline (−2.3; #492/#478) |
| street_prefix / suffix | 0 → 64.9 / 48.8 | — | ◐ exists at P≈100; ceiling measured (#492) |
| unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. | 92.3 → 90.6 (real-OOD) | — | ✅ retained (stated −1.7) |
| US micro | 80.2 → 84.8 | — | ✅ |
| DE native localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. | — | — | ✅ 90.9 (beats PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. 85.9) |
| po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. / cedexCEDEX (Courrier d'Entreprise à Distribution Exceptionnelle). A French postal routing for high-volume business mail: a CEDEX code delivers directly from a sorting centre, bypassing the local post office. A common negative-space format Mailwoman must parse. | starvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). | starvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). | ⏳ ride the next consolidation-class run (#492 rider) |
| intersection_a/b | 0 | — | ⏳ evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-first (#487) |
Campaign status
| Lever | status |
|---|---|
| unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. | ✅ banked (v4.1.0), retained at 90.6 |
| affix | ✅ exists (0 → 64.9/48.8, P≈100); solo-level stability requires architecture (#492) |
| countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. | ✅ banked (89.8, gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. soft anchoranchor inferenceA technique where structured knowledge (postcode locations, gazetteer place names) is injected into the model as soft input features — not as deterministic overrides. The model still decides the final labels, but the anchor signal biases it toward correct admin tags.; over-fire 0) |
| consolidation v1.0 | ✅ SHIPPED as v4.2.0 — the flag-plant, with stated re-baselines |
| FR venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. | ⏳ #330 — the next trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. lever |
| intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model. | ⏳ #487 evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-first |
| architecture escalation | 📋 #492 — operator GO required |
| arbitrationarbitrationA pipeline stage that compares rule-based (v0) and neural classifier output, resolving disagreements via a policy registry. Built and merged but not promoted — the coordinate gate showed label-F1 gains came at the cost of worse geocoding. | 📋 #478 — the post-parity capstone: converts every v0-only cell above into both |
Emitted night-10, 2026-06-10. Gate provenance: canonical config bars; re-baselines stated in the ship-gate doc and the night plan, never silently.