The harness is v0's home turf — reframing the scoreboard (2026-05-30)

TL;DR. The 415-assertion harness-v0-neural suite — our de-facto neural "promote bar" — is a port of the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser + addressit corpora, the same lineage the v0 rule parser was built from. v0 scores 93.7% on it almost tautologically; the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' scores 19.8%. Chasing a 25% neural score on a suite where neural is structurally disadvantaged is the wrong hill. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. end-to-end metric (address to correct WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. place) is the honest scoreboard, and there the same v0.7.2 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' already wins: neural 68.9% vs v0-via-adapter 63.7% Place-Match Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision.. This doc records the evidence (all numbers re-verified from clean runs) and proposes the metric reframe.

The numbers (v0.7.2, deterministic re-run, 2026-05-30)

scripts/harness-v0-neural.ts --tests mailwoman/test --falsehoods data/eval/falsehoods --postcode-repair against the v0.7.2 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (output-v072-intersection, tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. v0.6.0-a0):

Parser	Pass	Rate
v0 (rule-based)	389 / 415	93.7%
Neural (fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.)	82 / 415	19.8%
Neural tree structurally valid (#37)	391 / 415	94.2%

Outcome cross-tab (sums to 415, the integrity check):

Category	Count
Both pass	72
v0 only	317
Neural only	10
Both fail	16

Read the two rows that matter together: v0-only is 317, both-fail is 16. Of the 333 cases neural fails, 317 (95%) are cases v0 handles fine. Only 16 are hard for both parsers. The neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is not failing on intrinsically hard addresses — it is failing on addresses that are easy for a rule parser tuned on this exact corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus..

Why that is expected, not alarming

data/eval/external/README.md already documented the lineage problem for the external arenasarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.; it applies to the harness itself. The mailwoman/test/*.test.ts files the harness extracts assert(input, ...expected) calls from are, on inspection, a port of PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser + addressit — and our v0 parserrule-based classifierMailwoman's legacy v0 parser — a library of deterministic token classifiers (house number, street suffix, postcode, place name, etc.) composed by priority. Now primarily used for corpus labeling, fallback classification, and arbitration diagnostics. is itself PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-derived. So:

• The expected labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. encode PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor./addressit's segmentation conventions.
• v0 reproduces them by construction, hence 93.7%.
• The neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learned a different (often defensible) segmentation and is graded as wrong whenever it disagrees with the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. convention.

The matcher is already lenient — expectedMatchesActual passes on substring containment in either direction (harness-v0-neural.ts) — so pure over- or under-spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. boundary errors do not cause failures. The failures that remain are genuine disagreements: a tag the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' did not emit, or a value that is neither equal to nor a substring of the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-expected value.

The dominant failure shape: under-segmentation on non-canonical input

Spot-checking the failing cases (the dominant pattern across the 317 v0-only wins): the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' collapses adjacent fields on inputs that drift from the clean, comma-delimited US trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. distribution. Representative real failures:

• 6000, NSW, Australia — expected postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality./countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.; neural tagged 6000 as house_number and NSW as localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. (AU postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-leading line).
• Unit 12/345 Main St — expected unit_designator/unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise./house_number/streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.; neural produced streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. Unit 12, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. Main St (AU unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise.-slash format).
• U 12 345 Main St — neural produced only localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. U, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. 12.

The pattern is consistent: when the surface form drifts from the canonical comma-delimited US template, the segmentation degrades — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' leans on delimiter/format cues it saw in trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. rather than on tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. semantics.

The honest scoreboard: resolver end-to-end

The product goal is address to correct place plus coordinates, not match PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s component spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. The Direction-C resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. (scripts/eval/resolver-eval.ts, 2406 WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.-bootstrap rows, custom gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.) measures that directly. Numbers below were re-verified two ways: the script's own output and an independent recompute from the raw per-row sidecar — they agree to the decimal.

baseline	canonical	perturbed	all
neural-only	77.1%	64.8%	68.9%
v0-via-adapter	69.5%	60.8%	63.7%
arbiter (pick higher resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. score)	76.9%	69.5%	72.0%
oracle (either correct)	79.4%	77.1%	77.9%

On the metric that matches the product, the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' already beats v0 (+5.2pp Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. overall), and the old "route clean inputs to v0" thesis is dead — neural wins clean too (77.1 vs 69.5). The arbiter (dual-parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates., pick the higher-resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-confidence result) adds +3.1pp overall and +4.7pp on perturbed/noisy input over the best single parser. Coordinate error is p50 0 km / p90 1090 km for neural-only (all 2406 rows resolved).

Reframe (the proposal)

1. ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. end-to-end Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. is the headline metric. It matches the product, is lineage-neutral, and is where the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s value actually shows.
2. The harness-v0-neural suite is a regression gate, not a promotion bar. Keep running it — a drop signals a real regression — but stop treating "25% neural on a v0-lineage suite" as the goal. It asks the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' to imitate PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s conventions on PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s own corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus..
3. CoveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present., measured on lineage-neutral sets, is the lever. The under-segmentation on non-canonical formats (above) is the real gap, and it shows up on the perturbed resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. subset and the external arenasarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. — both lineage-neutral — not just the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. harness.

This reframe unblocks two stalled cycles (v0.6.x held after three recipe iterations; v0.7.x calibration null on the harness): both were optimizing a lineage-biased metric. It is consistent with the v0.7.2 evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.'s own conclusion ("the path to 25% is coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. and the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon., not another recipe tweak") and with project-neural-vs-v0-capability-map.

Next model experiment (one pre-registered hypothesis)

Continue-train from the v0.7.2 checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. with interleaved comma-free renders (house street locality region with the delimiter dropped), roughly 15% of batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU., low LR, capped steps, early-stop on the harness, with a held-out comma'd-only forgetting tripwire. Hypothesis: the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' leans on a comma cue that real inputs drop (339 Bedford Ave Brooklyn); teaching delimiter-invariance should move localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-missing and streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-absorbs-localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. together. Pre-registered revert: if post-train harness is below 22% and resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. does not improve by at least 1.5pp, revert — the lineage-mismatch conclusion stands.

Provenance note

These numbers were produced during a session in which the local tool-output display layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. had earlier been corrupting/duplicating command output. After the environment was restarted and verified clean (a fixed string hashed repeatedly returned a stable digest; a 17-command parallel batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. ran with no cancellations), both evalsevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. were re-run from scratch. The harness cross-tab sums to 415, and the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. table reproduces from an independent recompute of the raw sidecar — the figures here are trustworthy. Per-case forensic breakdowns that could not be cross-checked are deliberately omitted rather than reported at false precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1..