v0-vs-Neural harness — 2026-05-28

The neural parser has never been measured against the legacy rule-based pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.'s hand-tuned acceptance criteria. The 4561-entry golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. at data/eval/golden/v0.1.2/ measures the neural on what it was trained for; the 376 assertions in mailwoman/test/*.test.ts measure the rule-based parser on what it was hand-tuned for. This evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. bridges the two.

Per DeepSeek consult turn 6: without this harness, v0.6.2's corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. augmentation is "try something and hope." With it, augmentation becomes "fix the specific assertions that fail."

Setup

• Harness: scripts/harness-v0-neural.ts extracts every assert(input, ...expected) call from mailwoman/test/*.test.ts via TypeScript AST (376 assertions across 30 files spanning 20+ localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.), runs each input through BOTH createAddressParser() (v0 rule-based) and NeuralAddressClassifier (v0.6.0 + admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places). + morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition.), and reports per-file / per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. pass rates plus a JSON sidecar for downstream clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity..
• Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': v0.6.0 default (model-v060-step-100000-int8.onnx)
• Admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places).: fst-en-us.bin
• Morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition.: built in-process from libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. street_types.txt dictionaries (1,707 canonicals / 3,763 variants after length-3 filter)
• Comparison semantics:
  • v0: vitest toEqual strict deep-equality, position-by-position against ranked solutions (matches the existing test semantics — pass only if every expected solution deep-equals solutions[i].classifications).
  • Neural: flatten AddressTree to Map<ComponentTag, string>, fold neural-only tags (street_prefix + street + street_suffix → street; intersection_a + intersection_b → street as two values), then check if any expected solution matches the resulting record (substring containment in either direction allowed).

Result

Parser	Pass	Rate
v0 (rule-based)	376	100.0%
Neural	54	14.4%

Category	Count	Rate
Both pass	54	14.4%
v0 only	322	85.6%
Neural only	0	0.0%
Both fail	0	0.0%

Zero neural-only wins. Every assertion the neural passes is also passed by v0. The rule-based pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. strictly dominates the neural pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. on its own test suite.

Per-file breakdown

File	Total	Neural %	Comment
address.usa.test.ts	73	23%	Heaviest trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. distribution; still 56/73 missing
intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model..test.ts	65	0%	Total failure mode; Main St & 5th Ave-style inputs
functional.test.ts	34	3%	Cross-cutting cases
address.fra.test.ts	33	24%	Some French in trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.
address.nld.test.ts	22	9%	NL compact addresses largely unhandled
address.nzd.test.ts	22	0%	NZ format
addressit.usa.test.ts	21	81%	Autocomplete-style — best performer
address.deu.test.ts	17	0%	German format entirely missed
place.fra.test.ts	13	0%	French place lookups
addressit.aus.test.ts	11	64%	Australian autocomplete
address.aus.test.ts	9	0%	Australian unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. notation (Unit 12/345)
address.nor.test.ts	9	0%	Norwegian
address.prt.test.ts	8	0%	Portuguese
address.pol.test.ts	6	0%	Polish
venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label..usa.test.ts	6	0%	VenuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.-only inputs
address.rom.test.ts	5	20%	Romanian
address.swe.test.ts	4	0%	Swedish
compound_street.test.ts	4	0%	Compound streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. tests
address.cze.test.ts	3	0%	Czech
address.gbr.test.ts	3	0%	UK format
(other localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.)	1-2 each	0%	bra, esp, hrv, ind, svk, transit
libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it..test.ts	1	100%	Single passing fixture

Failure clusters

The 322 v0-only-passes split into four structural categories:

1. Tokenization issues with non-ASCII

Neural tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. + BIO decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. garbles multi-byte characters. Examples:

• Korunní 810, Praha → neural: {locality: ["Korunn"], region: ["ha"]} — ní is split mid-character
• Rua Raul Leite Magalhães, 65, Tapiraí - SP, 18180-000, Brazil → neural: {street: ["es"], region: ["zil"], venue: ["Rua Raul Leite Magalh"]} — Portuguese diacriticsdiacriticAn accent mark that modifies a letter (é, ñ, ç). Address normalization must fold diacritics for matching without discarding the information a user typed. destroy spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. boundaries

This is a tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses./encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). issue, not a schema issue. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' can't recover when SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. piecesECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). don't reassemble to the original characters.

2. Schema gap: unit_designator

v0 has both unit and unit_designator — for Unit 12/345 Main St, expected is {unit_designator: ["Unit"], unit: ["12"], house_number: ["345"], street: ["Main St"]}. The neural schema only has unit. Every Australian unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise.-notation test fails on this gap alone:

• Apartment 12/345 Main St → expected unit_designator: ["Apartment"] but neural has no tag for it
• U 12 345 Main St → same problem with U
• Lot 12/345 Illawarra Road... → 9 of 22 NLD/AUS tests fail on this single missing tag

This is solvable by adding unit_designator to the StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 3 schema (and the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.'s adapters), but it's a schema change requiring retraining.

3. House-number / street boundary

The neural parser keeps reading streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. and house_number as one spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. German examples:

• Am Nordkanal 11, 47877 Willich → expected street: "Am Nordkanal" + house_number: "11"; neural: street: "Am Nordkanal 11"
• Am Falkpl. 5, 10437 Berlin → expected street: "Am Falkpl."; neural: street: "Am Falkpl" (period dropped) — close but not equal under strict comparison

The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' never saw enough German addresses to learn the prepend-house-number pattern. v0's HouseNumberClassifier hard-codes the heuristic; the neural has to learn it from data.

4. Intersections (0/65)

Every intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model. test fails. Main St & 5th Ave → expected {street: ["Main St", "5th Ave"]} (two streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. values); neural produces single-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. output or routes one side into a non-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. tag. This is the failure mode the falsehoods doc flagged. v0 has a dedicated IntersectionClassifier + CompositeIntersectionClassifier; the neural has intersection_a/b BIO tags but the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data appears to undersample intersectionsintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model. relative to v0's hand-tuned coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present..

What this means for v0.6.2

This is the "honest assessment" the postmortem called for. The implications:

1. LocalelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. is the dominant gap. US, FR, and a slice of AUS are in distribution; everything else is structurally untaught. v0.6.2's corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. augmentation has to expand beyond US-only synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels..
2. Schema additions are needed before retraining helps Aus/UK unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. notation. The unit_designator gap can't be papered over by trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-data tweaks.
3. IntersectionsintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model. need targeted corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. work. v0.6.0's synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. taught streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. decomposition but didn't include intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model. patterns at all.
4. TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. is suspect for non-ASCII. Czech / Portuguese / German diacriticsdiacriticAn accent mark that modifies a letter (é, ñ, ç). Address normalization must fold diacritics for matching without discarding the information a user typed. are destroying spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. boundaries. Needs investigation independent of the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. path.
5. Strict equality might be the wrong bar for the neural parser. v0's tests assert exact structural equality including punctuation. The neural pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. currently doesn't preserve tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-level punctuation. A looser equivalence (Am Falkpl. ≈ Am Falkpl) would change the picture for cluster #3 specifically; whether that looser bar is acceptable is a product decision.

Falsehoods extension

The harness also accepts a --falsehoods <dir> argument for JSONL files of structured rows. data/eval/falsehoods/streets.jsonl lands the catalog from falsehoods-streets.md as 22 explicit test rows — Piccadilly, rue de Rivoli, Plein 1944, Avenue Road, Gondel 2695, the A1-as-composite-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., dependent streetsdependent streetA secondary street name required for delivery in some postal systems (notably Royal Mail), as in '6 Elm Avenue, Runcorn Road, Birmingham' where Runcorn Road is the dependent street., Japanese block addressing, rural routes, Mannheim grid. Each row carries falsehood (the category) and expected_failure (whether this is a known gap rather than a regression vector).

With the falsehoods rows added the picture is:

Parser	Pass	Rate
v0 (rule-based)	385 / 398	96.7%
Neural	56 / 398	14.1%

Category	Count	Rate
Both pass	54	13.6%
v0 only	331	83.2%
Neural only	2	0.5%
Both fail	11	2.8%

Two neural-only wins appear once the falsehoods are in play — cases where the neural parser succeeds and the rule-based pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. fails:

1. 8 Seven Gardens Burgh, WOODBRIDGE, IP13 6SU — number-in-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-name. v0's HouseNumberClassifier grabs the leading 8 then mis-tags the rest; neural reads it as a single streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. correctly.
2. R 5, 6-13, D-68161 Mannheim — grid address with no streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.. v0's classifiers force a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-shaped reading that doesn't fit; neural correctly produces just localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. + postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one..

The pattern: neural is sometimes BETTER on formats it has never seen, because v0's hand-tuned rules actively misclassify them. The "ambiguous-but-common" cases (rue de Rivoli, Broadway, place de la Concorde, Avenue Road) is where v0's rules dominate — neural lacks the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. distribution.

V0 also fails 13 of 22 falsehoods — Piccadilly, Hauptstraße 5, Plein 1944, Gondel 2695, 6 Elm Avenue, Japanese block addressing, and rural routes are unsolved by both parsers. The falsehoods catalog is the right capability-boundary regression suite for any future release.

Reproducing

node scripts/harness-v0-neural.ts \
  --tests mailwoman/test \
  --model $MAILWOMAN_DATA_ROOT/models/quantized/model-v060-step-100000-int8.onnx \
  --tokenizer $MAILWOMAN_DATA_ROOT/models/tokenizer/v0.6.0-a0/tokenizer.model \
  --model-card neural-weights-en-us/model-card.json \
  --admin-fst $MAILWOMAN_DATA_ROOT/wof/fst-per-locale/fst-en-us.bin \
  --out-json /tmp/harness-full.json \
  > /tmp/harness-full-report.md

Total runtime: ~5 seconds for 376 assertions.

See also

• Street-supplement architecture — the design context
• Layer 1 morphology FST eval — preceding evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. that established the decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs.-only fix is insufficient
• Falsehoods about street names — the edge cases the harness's falsehoods row source captures
• 2026-05-28 night-2 postmortem — postmortem that triggered this assessment