Skip to main content

v0.6.2 — Stage 3 corpus rebalance eval

The v0.6.2 release fixes the v0.6.1 dependent_locality regression via the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. rebalance recipe documented in street-supplement-architecture.md. This doc captures the empirical results.

This page is a draft template — the TBD fields land when the v0.6.2 100K evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. (scripts/eval-v062-checkpoint.sh) completes.

Setup​

  • Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': v0.6.2 (model-v062-step-100000-int8.onnx)
  • TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.: v0.4.0 base + synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. (122K rows; 112K US + 10K DE/GB; 35% venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.-adversarial)
  • Source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. delta from v0.6.1: synth-street 2.0 → 0.5; synth-no-street 1.0 (new)
  • TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.: v0.6.0-a0
  • InferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.: admin FSTadmin FSTA finite-state transducer encoding Who's On First admin place names (countries, regions, localities, postcodes) for fast gazetteer lookup and emission priors at inference time. Ships to the browser (~9 MB for 94K US admin places). fst-en-us.bin + morphology FSTmorphology FSTA finite-state transducer encoding street-typing affixes (Avenue, rue, Calle) from libpostal dictionaries, for morphological street recognition. (libpostallibpostalAn open-source C address parser used by Pelias. Mailwoman's rule-based v0 and neural classifier supersede it. street_types, length-3-filtered, 1,707 canonicals)
  • Golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals.: data/eval/golden/v0.1.2/ (4,561 entries)
  • v0-vs-neural harness: mailwoman/test/*.test.ts (376 v0 assertions) + data/eval/falsehoods/ (39 falsehoods rows) = 415 total
  • Gate baseline: v0.6.0 default per-tag JSON (/tmp/eval-v060-true-baseline.json)

Headline numbers (TBD)​

Metricv0.6.0v0.6.1v0.6.2Δ vs v0.6.0
Exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness. (golden)21.1%18.8%TBD—
localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.39.7%31.1% (-8.6pp ❌)TBD—
dep_locality hallucinations01066 ❌TBD (target ≤100)—
street_suffix hallucinations0198TBD—
v0-vs-neural harness pass14.4%(≈14.4%)TBD—
Pre-publish 2D evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. gate(baseline)FAIL ❌TBD (target PASS)—

Per-tag breakdown (TBD)​

Tagv0.6.0 recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.v0.6.2 recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.Δv0.6.0 hall.v0.6.2 hall.Δ
localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.39.7%TBDTBD503TBDTBD
regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.65.0%TBDTBD117TBDTBD
postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.76.0%TBDTBD145TBDTBD
streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.27.9%TBDTBD46TBDTBD
house_number79.0%TBDTBD381TBDTBD
venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.29.2%TBDTBD86TBDTBD
countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.24.5%TBDTBD101TBDTBD
dependent_locality0% (0 hall.)TBD (target ≤100 hall.)—0TBDTBD
po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise.51.9%TBDTBD4TBDTBD
street_prefix0%TBDTBD0TBDTBD
street_suffix0%TBDTBD0TBDTBD

Per-locale harness breakdown (TBD)​

The harness's 415 assertions cluster by localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.. v0.6.0 baselines:

LocalelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.Totalv0.6.0 passv0.6.2 passΔ
usa7317 (23%)TBDTBD
intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model.650 (0%)TBDTBD
functional341 (3%)TBDTBD
fra338 (24%)TBDTBD
nld222 (9%)TBDTBD
nzd220 (0%)TBDTBD
deu170 (0%)TBDTBD
aus90 (0%)TBDTBD
nor90 (0%)TBDTBD
(smaller localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.)—mostly 0%TBDTBD

Gate verdict (TBD)​

[eval-gate.ts output here]

Decision: TBD​

Apply DeepSeek turn 7's promotion decision tree:

GateHarness passAction
Pass≥ 25%Promote v0.6.2 to default on HF
Pass15-24%Ship as experimental — do NOT promote
Pass≤ 14.4%HOLD — sidegrade not release
FailanyHOLD

Verdict: TBD

Notes from the training run​

(Filled in incrementally as the run progressed.)

  • Throughput: ~16 steps/s on A100 80GB PCIe (vs ~6 steps/s for the diagnostic on A100-SXM4-40GB). 100K steps ≈ 1h45min wall-clock.
  • Lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. trajectory: warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup. → step 1000 train_loss=0.286 → step 2000 train_loss=0.097 (first evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.: val_loss=0.333, macro_f1=0.371) → step 8000 macro_f1=0.410, localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.=0.862, regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.=0.725, streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.=0.984, house_number=0.999, postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.=1.000 → step 12000 macro_f1=0.413.
  • No NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.. CE-only trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. is stable (as expected — the bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU.-NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol. issue from the postmortem only affected CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality., which v0.6.2 leaves disabled per the NaN protocolNaN protocolThe NaN-recovery discipline: stop, change one variable, retry, document the hypothesis — never adjust two knobs at once during recovery, or you can't tell which change fixed it.).
  • Early-evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. gate at step 20K: TBD (dep_loc hallucination ceiling 100; abort + bump synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. size if exceeded).

See also​