NPPES NPI dedup benchmark (#617)
Generated by scripts/record-matcher/nppes-dedup-benchmark.ts. Sample: 300 CA NPIs with ≥1 alternate name → 832 records (2.8/NPI) from real registry + other-names + mailing-vs-practice variation. Matcher run BLIND to the NPI, EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-trained (labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free); geocoded 100.0% of addresses. The NPI is held-out ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy..
The comparison-model levers (each toggled on, at the default threshold)
| modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 | ARI | over-merged |
|---|---|---|---|---|---|---|
| baseline (address-key + distance) | 42.6% | 47.8% | 45.0% | — | 0.449 | 42 |
| + inverse-address-frequency (#617) | 45.2% | 78.2% | 57.3% | +12.3pp | 0.572 | 34 |
| + collapsed spatial signal (A1, #625) | 46.9% | 78.2% | 58.6% | +1.3pp | 0.585 | 36 |
Inverse-frequency weighting uses the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (3,317,267 distinct addresses over 9,260,504 providers) to down-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. a crowded shared address; collapsing the redundant address-key + distance comparisons into one spatial signal (A1) removes the double-count that let a shared address over-vote a disagreeing name. Across the levers, F1 45.0% → 58.6% (+13.6pp).
With all levers on, across the link threshold (the secondary lever)
| link threshold (bits) | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ARI | clusters | over-merged |
|---|---|---|---|---|---|---|
| 0 (default) | 46.9% | 78.2% | 58.6% ⭐ | 0.585 | 291 | 36 |
| 4 | 44.1% | 47.8% | 45.9% | 0.458 | 456 | 38 |
| 8 | 44.7% | 42.8% | 43.7% | 0.436 | 477 | 38 |
| 12 | 44.9% | 38.2% | 41.3% | 0.412 | 507 | 35 |
| 16 | 45.5% | 38.0% | 41.4% | 0.413 | 512 | 33 |
| 20 | 46.5% | 38.0% | 41.8% | 0.417 | 515 | 32 |
Best F1 is at the default threshold (58.6%): raising it only trades recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. away faster than it buys precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.. So the threshold knob alone can't separate co-located distinct providers — the over-merge is structural, in the comparison modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', not the cutoff.
Shape + where the errors are (at the default threshold)
- records: 832 · true entities (NPIs): 300 · recovered clusters: 291
- candidate pairs blocked: 10288
- Over-merge (precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.): 36 clusters fuse ≥2 distinct NPIs (largest fuses 8; 242 records) — co-located providers sharing a clinic / billing address, linked by address agreement despite different names.
- Under-merge (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.): 73/300 NPIs split across >1 cluster — records at distant addresses (mailing vs practice) or with strong name drift the score didn't bridge.
Reading
The geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' foundation works: 100.0% of addresses placed, blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. + clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. clean — the geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. (the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor./Nominatim-can't-do-this part) is not the bottleneck. The comparison-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' levers above moved F1 45.0% → 58.6% (+13.6pp): inverse-frequency weighting restores full weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. to a rare shared address (stitching a provider's name-drifted records together — mostly recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.) while down-weighting a crowded one, and the collapsed spatial signal (A1) drops the address+distance double-count. What remains is precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. / over-merge — 36 clusters still fuse distinct co-located providers, because even one down-weighted spatial agreement can outvote a disagreeing name. A name/org/phone corroboration gate (A2/A3) was investigated and does NOT beat this baseline on NPPES: phone is an unreliable secondary identifier here (shared institutional switchboard lines), so it over-links via blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. and falsely corroborates co-phone distinct providers — a documented negative (#625). The real over-merge lever is therefore average-linkage clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. (A4) replacing fragile connected-components, plus a more reliable secondary identifier (authorized-official, taxonomy). Config dominates the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (the pre-registered finding) — tracked as #625 / the auto-tuning + selective-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' work (#602 / #603).
NPI-as-truth is conservative: a cluster fusing two NPIs is a candidate "same entity, two NPIs" surfaced for review, not an adjudicated error; an NPI split across distant addresses is geo-first behaving correctly, counted here as a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. miss. We resolve and report; interpretation is the consumer's.