Punctuation-stress + paired-delimiter eval — the span proposer doesn't earn its revival (2026-06-14)

Closes the measurement half of #518. The question was narrow and gated: our evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. surfaces (OA/NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses./golden) are government data, punctuation-poor by construction, so neither engine had ever been graded on the quadrant where real user input lives — quoted venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. names, parenthetical annotations, c/o lines, unbalanced delimiters. #518 said: measure the class first, and only revive the abandoned StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2.7 paired-delimiter spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. proposer if the numbers say it earns its keep. They don't.

What was measured

data/eval/external/punctuation-stress.jsonl — 200 hand-curated + arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.-mined rows across 11 classes (quoted venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label., parenthetical annotation, parenthetical component, bracketed, c/o, dotted abbreviations, hyphenated names, slash/fraction, apostrophe, "mixed-hard", and unbalanced delimiters). Gold convention: delimiters are excluded from component values unless postal-meaningful (the # in a unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise., the dots in P.O.); apostrophes and hyphens inside a tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. are kept (O'Brien, Winston-Salem). Conventions in data/eval/external/punctuation-stress.README.md.

Scored by scripts/eval/score-punctuation-stress.ts (per-component exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness., case-insensitive, on each engine's own vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. view of identical gold). Each row also measures parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. survival — a thrown parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. fails every component, which is exactly what the unbalanced-delimiter rows exist to catch.

• v0 — the legacy rule parser (createAddressParser), deterministic, no modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'.
• neural — the shipped int8 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (model-v140-step-40000-int8, package v4.6.0; tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. v0.6.0-a0), full ship config (anchor + gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. + convention auto + punctuation-gap bridge). Graded on the folded gold view (affixes joined into street) for an apples-to-apples headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. with v0.
• neural + spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. proposer — the StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2.7 paired-delimiter proposer (--span-proposer, default-off, NOT ship config), at three bias settings.

Results

class	v0	neural	+SP (default)	+SP (bias 2)	+SP (bias 4 / ann 4)
apostrophe	89.0	80.8	80.8	80.8	80.8
bracketed *	80.3	85.6	85.6	85.6	76.5
care-of	60.8	72.5	72.5	72.5	72.5
dotted	82.3	74.0	74.0	74.0	74.0
hyphen	86.9	80.8	80.8	80.8	80.8
mixed-hard	58.2	72.7	72.7	70.0	66.4
paren-annotation *	78.4	87.5	87.5	87.5	68.2
paren-component *	76.5	82.4	82.4	82.4	82.4
quoted-venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. *	75.3	74.1	74.1	74.1	72.8
slash	71.8	61.5	61.5	58.1	59.8
unbalanced *	68.3	82.5	82.5	82.5	84.1
overall	75.7	77.3	77.3	76.6	73.4
parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. deaths	2	0	0	0	0

* = paired-delimiter classes (the proposer's target). Component accuracy %, 200 rows, folded gold.

Verdict: the span proposer does not earn its revival (as implemented)

At every bias tested, the StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2.7 spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. proposer is a no-op or a regression on the paired-delimiter classes it was built for:

• Default (no bias): exactly zero effect — +0.0pp on every class. As wired, the proposer contributes no bias unless --sp-bias is set, so the shipped default does literally nothing.
• Gentle (bias 2): −0.7pp overall, no help — the four paired-delimiter classes are unchanged; the only movement is slash and mixed-hard going down.
• Strong (bias 4 / annotation 4): −3.9pp overall, actively harmful — paren-annotation collapses 87.5 → 68.2 and bracketed 85.6 → 76.5, because the annotation bias pushes the wrong direction: it merges the delimited content into the adjacent spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. (Wallaby Way (rear entrance, Water St [SE corner) instead of stripping it.

So the abandoned Chevrotain-style proposer isn't a drop-in win waiting to be switched on. Its annotation semantics need a fix (the bias has the wrong sign — it should suppress, not absorb), not a parameterparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. sweep. Reviving it as-is would regress the very class it targets. Recommendation: do not revive on these numbers; if pursued, treat it as new design work, not a flag flip. (Caveat: three bias configs are not an exhaustive sweep — but a featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. that's a no-op at default and netneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network.-negative at the two non-trivial settings has not cleared the bar #518 set.)

The finding that matters more: neural already wins, and its failure mode isn't v0's

The headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. is the real takeaway:

• Neural beats v0 overall (77.3 vs 75.7) and is categorically more robust — 0 parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. deaths vs 2. On the unbalanced-delimiter rows that exist to test "degrade, don't die," neural degrades gracefully (82.5%) while v0 throws on the malformed input.
• The two engines fail differently. v0 shatters on quotes and poisons neighbors: "Big Company HQ" loses the venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. entirely (the quote is a hard field wall — the fieldsFuncBoundary TODO admits it), and 1600 Pennsylvania Ave NW (The White House), Washington poisons the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. to White and shifts the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. to Washington. Neural instead over-extends spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.: it absorbs the next component or the delimiter run into the current spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. — Sydney NSW (localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. eats the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.), Oxford OX1 4DB (localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. eats the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.), Rue du Bac (escalier B (streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. eats the unbalanced paren).
• v0 still wins the within-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. classes — apostrophe (89.0 vs 80.8), dotted (82.3 vs 74.0), hyphen (86.9 vs 80.8), slash (71.8 vs 61.5). Its rules handle 123 1/2, O'Brien, St. precisely; neural wobbles on fractional house numbershouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. and absorbs trailing tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words..

The implication for the roadmap: the highest-leverage punctuation-stress lever is not a new spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. proposer — it's reducing neural's spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. over-extension at delimiters (a boundary/decode problem, kin to the Saint-AlbansBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. fragmentation and the #555 locateSpan over-run). That's where the paired-delimiter rows actually break, and it's a sharper, cheaper target than reviving StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2.7.

Reproduce

# v0 (deterministic, no model)
node scripts/eval/score-punctuation-stress.ts --engine v0
# neural ship config (folded gold, head-to-head)
node scripts/eval/score-punctuation-stress.ts \
  --engine neural --model neural-weights-en-us/model.onnx --fold-gold
# + span proposer (the revival question)
… --span-proposer                       # default: no-op
… --span-proposer --sp-bias 2           # gentle: −0.7pp
… --span-proposer --sp-bias 4 --sp-ann-bias 4   # strong: −3.9pp, merges annotations

Caveats

• Folded-gold view (affixes → street) for the headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.; per-engine vocabularies graded on their own view of identical gold (stated in the scorer header).
• Neural numbers are the shipped int8 v140 / v0.6.0-a0 tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.. Don't compare these absolute figures across tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. versions — the headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-to-headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. and the spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-proposer deltas are what matter here.
• Three spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-proposer bias configs, not an exhaustive sweep. The verdict is "doesn't clear the #518 bar as implemented," not "no parameterization could ever help."