Leakage-split F1 — openaddresses-us-sample.jsonl
Per-tag F1 split by corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-held-out geography (VT/WY/ND) vs in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. geography. A large in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. − held-out gap = the headline F1 is partly memorizationoverfittingWhen a model memorizes quirks of its training set instead of learning general patterns, so it scores well in training but poorly on new data. Guarded against with held-out evals and regularization. (#371).
- held-out rows: 1428 · in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. rows: 8572 · modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.':
neural-weights-en-us/model.onnx(v4.0.0)
| tag | held-out F1 | in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. F1 | gap (in − held) |
|---|---|---|---|
| localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. | 1.000 | 0.987 | -0.012 |
| regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. | 1.000 | 0.999 | -0.001 |
| postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. | 1.000 | 0.996 | -0.004 |
| macro | 1.000 | 0.994 | -0.006 |
Per-state macro-F1 (difficulty confound check)
| stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. | n | macro-F1 | held-out? |
|---|---|---|---|
| MT | 4284 | 0.979 | |
| DC | 4287 | 0.992 | |
| IA | 4287 | 0.996 | |
| SD | 4284 | 0.998 | |
| CA | 4287 | 1.000 | |
| VT | 4284 | 1.000 | ✅ held-out |
| IL | 4287 | 1.000 |
Read
DeepSeek (2026-06-07) flagged geographic train/test leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits. as the top risk to our headline F1: the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' trains on the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. (tigerTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data./BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure./WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.), which covers the same streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. and localities OA tests, so component recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. could be partly memorizationoverfittingWhen a model memorizes quirks of its training set instead of learning general patterns, so it scores well in training but poorly on new data. Guarded against with held-out evals and regularization.. The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. holds out specific US geography (VT/WY/ND) from trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., so OA rows there test places the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' never saw.
The result refutes that hypothesis on the tags OA can grade. Held-out Vermont scores 1.000 macro-F1 against 0.994 in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., so held-out is if anything marginally easier. The per-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. spread tells the same story from another angle: Montana (in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.) is the worst at 0.979 while Vermont (held-out) ties California and Illinois at the top. The ordering tracks intrinsic stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. difficulty, not trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. exposure. The shipped en-US modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality./postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. recognition generalizes to unseen geography, and the near-perfect numbers also confirm that these three tags on clean, canonical US addresses are essentially a solved problem.
One caveat carries the whole result, so it's worth stating plainly. This only tests locality, region, and postcode, because that's all OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. gold carries. LeakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits. would bite hardest on STREETstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. recognition, where memorizing a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. name is the obvious shortcut, and OA can't grade streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. at all. A complete leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits. check needs full-BIO corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. gold restricted to held-out geography, which is tracked as a follow-up on #371. And only Vermont survives in this US sample (Wyoming and North Dakota contribute zero rows), so the held-out signal is one stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. wide.
Reproduce: node scripts/eval/leakage-split-f1.ts --eval data/eval/external/openaddresses-us-sample.jsonl --held VT,WY,ND --out-md <path>.