v0.7–v0.8: the neural parser vs the rules baseline

When: 2026-05-29 → 2026-05-31.

This retrospective asks three questions. Is the neural parser ready to replace v0 (our TypeScript port of the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. rule engine)? If so, why does our own headline test suite say it's nowhere close? And when those two answers disagree, which one should we believe?

Well… it depends on how we're keeping score. On our in-house suite the rules parser wins in a landslide, 93.7% to 20.7%. On the scoreboard that matches the product — real addresses, resolved end to end — the neural parser comes out ahead, 97.3% to 95.8%. Both numbers are real; they're just counting different things.

This is the deep-dive companion to the blog post (Our parser fails 80% of our own tests). It pins down the numbers, the bias, and the forward plan, with enough detail to be useful in three months.

All neural numbers below are v0.7.2-class (the shipped default; its v0.8.1 retrain at the same recipe is statistically identical). TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. v0.6.0-a0 throughout, so cross-run comparisons are valid. v0 is the rule parser, scored through the same machinery.

Scoreboard 1 — the harness says neural is 4.5× worse

The regression gate is scripts/harness-v0-neural.ts: 415 tag-level assertions ported from the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. / addressit test suites. Current standings:

parser	pass	rate
v0 (rule-based)	389	93.7%
neural	86	20.7%
both pass	76	18.3%
v0 only	313	75.4%
neural only	10	2.4%
both fail	16	3.9%

On paper the rules parser wins in a walk. Then look at the per-file breakdown: v0 passes 100% of every functional file (usa, intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model., functional, fra, nld, nzd…). Naturally — v0 is our port of PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor., and these assertions are PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. / addressit's own expected outputs, so it passes by construction. The suite cannot, even in principle, show v0 being wrong on these cases; it encodes PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s tagging conventions as ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy..

So "neural scores 20.7%" measures one thing: how often neural disagrees with PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s exact tagging conventions — 75% of the time, on different tokenization of multi-word streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., different venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label./localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. boundaries, and the cases addressit happened to encode. It says nothing about whether neural is wrong. The harness is a faithful regression gate (it catches when a retrain breaks a convention we previously matched), but not a parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.-quality verdict. Treating it as one is the harness-lineage trap.

Decomposing the 20% — three unbiased arenas

To see where neural actually earns its keep, we score both parsers on three arenasarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. drawn from outside the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. lineage (scripts/eval/external-arenas.sh, all --symmetric-match):

arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.	n	v0	neural	both	neural-only	v0-only	both-fail
libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. (clean / canonical)	69	29%	16%	9%	7%	20%	64%
perturb (noisy / degraded)	398	39%	61%	32%	29%	7%	32%
postal (edge formats)	38	26%	8%	5%	3%	21%	71%

Three different stories:

• Clean, canonical input → rules win (29% vs 16%). libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it.-style strings are exactly what hand-tuned rules are built for. This also explains the harness: it is all canonical, PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-convention input, so it is the arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. where neural is weakest, sampled 415 times.

• Noisy, degraded, reordered input → neural wins, decisively (61% vs 39%) — and this is the largest arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. (398 cases), generated by perturbing golden addresses (dropped commas, abbreviations, reordering, casing). This is the closest proxy we have to real geocoder traffic, and it is the entire reason the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' exists: it generalizes where rules brittle-fail.

• Edge formats → both are bad, v0 marginally ahead (26% vs 8%). The edge_class breakdown of the postal arenaarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins.:

  edge_class	n	v0	neural
  canonical	14	36%	7%
  intl-format	7	43%	29%
  secondary-unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise.	7	29%	0%
  directional	2	0%	0%
  military-apofpo	3	0%	0%
  po-box	4	0%	0%
  rural-route	1	0%	0%

  PO boxesPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise., military APO/FPOAPO/FPO (Army Post Office / Fleet Post Office). US military postal addresses for personnel stationed overseas or aboard ship. They route through military postal hubs rather than geographic locations, so they break geographic parsing assumptions., and rural routes are 0% for both — neither parser was built or trained for them.

Scoreboard 2 — the resolver, on real addresses, end to end

The product is a geocoder, not a tag-matcher. The honest, non-circular metric is the OpenAddresses real-point eval: every row is a real US address with a real government lat/lon, and we score whether each parser, run through the same WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon., lands on the right localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.. On 10,000 rows:

parser	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. match	regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. match	resolved
neural	97.3%	99.9%	100.0%
v0 (PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.)	95.8%	99.5%	99.8%

On the metric that matches the product, neural beats the rules parser (+1.5pp localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., on real addresses). The north-star claim — "a parser superior to PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.'s regex/rules system" — is demonstrated here, not merely asserted.

(Coordinate accuracy is still admin-centroid tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous.: p50 2.4 km, p90 10.6 km. That is the geometry gap, not a parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. gap — addressed in the roadmap below.)

Where neural is worse at parsing — and why

Stripping out the harness bias, the real neural deficits are:

1. Clean canonical forms (libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. 29 vs 16). Rules encode the exact pattern; neural has no inherent edge there, and its documented overconfidence (≈81% of wrong predictions emitted at ≥0.9 confidence) plus reliance on comma/format cues can mis-segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context. an otherwise tidy string.
2. Edge / postal formats — PO boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise., military, rural route (0% both); secondary-unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. 0%. Why: under-represented in the synthetic trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., so never learned.
3. Non-US localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. On the harness the neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' scores 24% (FR), 23% (NL), 0% (NZ) against v0's 100%. Why: it is an en-us modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'; it was not trained for NZ/NL/FR conventions.
4. Concrete parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. bugs surfaced in the resolver failure analysis: multi-word localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. (Belle Fourche→Belle, Fort Pierre→Fort), DC directional quadrants (…Street Ne → tags Ne as a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.), and literal <Null> tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. in dirty input that v0 shrugs off.

Every one of these is corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-addressable — they are gaps in what the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' was shown, not architectural limits.

Where neural wins — and why we run it

Noisy real-world input (perturb 61 vs 39) and the product metric (97.3 vs 95.8 on real points). Real traffic is abbreviated, reordered, comma-light, misspelled — exactly where rules brittle-fail and a learned modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' generalizes. The 10 harness "neural-only" wins (up from ~2 historically) are the leading edge of the same effect.

The MLM detour — a clean negative result

We spent part of this window testing whether masked-language-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. of the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). would lift the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (attacking the overconfidence + format-reliance pathologies). A 40k-step fine-tune from a pre-trained encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). looked like a win (+4.8pp harness, a calibration edge). But running the decisive A/B at the full v0.7.2 recipe — both arms to the task ceiling, single variable = encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). init — the gains vanished:

metric	pretrained	scratch	Δ
resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. (10k)	97.3%	97.5%	−0.20 (0.9σ, tie)
harness	19.04%	20.72%	−1.69
wrong-pred conf p90	0.949	0.949	0.00

The 40k "gains" were an under-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. artifact: a pre-trained init is simply ahead before a scratch encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). catches up; given 100k steps it does. Per a pre-registered kill-point, we dropped pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.. v0.7.2 stays the default. The code is kept for a possible future larger modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — the negative result is specific to this 29M-param encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters)., 20k-step pre-train, en-us corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus..

What we'd do differently / lessons

1. Don't let a PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor.-derived suite judge a non-PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor. parser. A test suite ported from the system you're trying to beat scores that system at 100% by construction. Use it as a regression gate; judge quality on independent arenasarenaA standardized test set probing one capability: libpostal (clean canonical), perturb (noisy and degraded), postal (edge formats). Each arena answers a different question about where rule vs neural wins. + the product metric.
2. Pick the metric that matches the product. Per-tag agreement is not geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. accuracy. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.-on-real-points number is the one that tracks the metric that matters, and it tells the opposite story from the harness.
3. Pre-register kill-points. The MLM track stayed alive on a 40k mirage; the pre-registered "ship only if the gains hold at ceiling" rule is what killed it cleanly instead of inviting another speculative round.

Forward plan (DeepSeek turn-5, ranked)

1. StreetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-level geometry (TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data./Line + house-number interpolationinterpolationA geocoding technique that estimates a coordinate along a street segment based on the house number range. Used as the middle tier of Mailwoman's geocode cascade when exact address-point data is unavailable.). We win on localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. but coordinates are admin-centroid only (p50 2.4 km). This is the km → tens-of-meters jump that makes "production geocoder" literally true. Gate: p50 ≤ 200 m, p90 ≤ 1 km on a 1,000-point ground-truth set. ⚠ Requires geometry we don't yet have (our tiger.db is streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-name + ZIP only) — a new data pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package., not a wiring job.
2. ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. ranking depth — fix the same-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. wrong-town bug (Saint Albans, VT → St. Johnsbury) and harden the population/importance tiebreak. Cheap; lifts localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. ~97.3 → ~98%. Good first/parallel win.
3. Parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.-robustness corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. pass — multi-word localities, directional quadrants, noisy tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., the edge formats at 0%. Feeds the next train, closes the genuine neural deficits above.
4. Multi-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. — extend beyond en-us once the US geocoder is streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-level complete.