NPPES NPI dedup benchmark (#617)
Generated by scripts/record-matcher/nppes-dedup-benchmark.ts. Sample: 300 TX NPIs with ≥1 alternate name → 816 records (2.7/NPI) from real registry + other-names + mailing-vs-practice variation. Matcher run BLIND to the NPI, EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-trained (labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free); geocoded 100.0% of addresses. The NPI is held-out ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy..
The comparison-model levers (each toggled on, at the default threshold)
| modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 | ARI | over-merged |
|---|---|---|---|---|---|---|
| baseline (legacy: address-key + distance, levers OFF) | 47.5% | 40.5% | 43.7% | — | 0.436 | 39 |
| + inverse-address-frequency (#617, corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide) | 55.7% | 74.6% | 63.8% | +20.1pp | 0.637 | 36 |
| + collapsed spatial signal (A1, #625) | 55.8% | 74.6% | 63.9% | +0.1pp | 0.638 | 36 |
| + authorized-official discriminator (#625, full stack) | 54.5% | 74.6% | 63.0% | -0.9pp | 0.629 | 38 |
Inverse-frequency weighting uses the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (3,317,267 distinct addresses over 9,260,504 providers) to down-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. a crowded shared address; collapsing the redundant address-key + distance comparisons into one spatial signal (A1) removes the double-count that let a shared address over-vote a disagreeing name. The address-frequency + A1 baseline is F1 63.9% at the default threshold; the authorized-official discriminator is roughly neutral there (-0.9pp) but enables a higher cutoff — it holds recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. where the baseline alone collapses, reaching 64.7% at threshold 12 (below), the first config past the baseline (#625).
Out-of-the-box (zero-config resolveEntities(records), levers default-on #86): F1 44.5% on this sample — essentially the baseline. The default auto-computes the address-frequency table over the INPUT records, and this benchmark deliberately sub-samples 816 records, so that table is too sparse to carry the inverse-frequency signal (a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. statistic you can't synthesize from a slice). It's not a regression (≥ baseline, no over-merge added) — it's the honest floor when the input isn't corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-scale. Fed the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (the + inverse-address-frequency row above, what the CLI builds from the full source files) the SAME default reaches the 63.9% baseline. On a full-dataset dedup the input IS the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., so zero-config reaches the baseline on its own.
With all levers on, across the link threshold (the secondary lever)
| link threshold (bits) | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ARI | clusters | over-merged |
|---|---|---|---|---|---|---|
| 0 (default) | 54.5% | 74.6% | 63.0% | 0.629 | 305 | 38 |
| 4 | 55.7% | 74.6% | 63.8% | 0.637 | 309 | 37 |
| 8 | 56.4% | 74.6% | 64.2% | 0.641 | 311 | 36 |
| 12 | 57.2% | 74.6% | 64.7% ⭐ | 0.646 | 312 | 35 |
| 16 | 48.7% | 40.5% | 44.2% | 0.441 | 490 | 36 |
| 20 | 45.6% | 35.3% | 39.8% | 0.397 | 522 | 35 |
Best F1 64.7% (ARI 0.646) at threshold 12, vs 63.0% at the default — the threshold knob moves it 2pp. Higher thresholds trade recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. for precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1..
Shape + where the errors are (at the default threshold)
- records: 816 · true entities (NPIs): 300 · recovered clusters: 305
- candidate pairs blocked: 6915
- Over-merge (precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.): 38 clusters fuse ≥2 distinct NPIs (largest fuses 13; 208 records) — co-located providers sharing a clinic / billing address, linked by address agreement despite different names.
- Under-merge (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.): 87/300 NPIs split across >1 cluster — records at distant addresses (mailing vs practice) or with strong name drift the score didn't bridge.
Reading
The geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' foundation works: 100.0% of addresses placed, blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. + clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. clean — the geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. (the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor./Nominatim-can't-do-this part) is not the bottleneck. The comparison-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' levers reach the address-frequency + A1 baseline at F1 63.9% (+20.1pp over baseline): inverse-frequency weighting restores full weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. to a rare shared address (stitching a provider's name-drifted records together — mostly recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.) while down-weighting a crowded one, and the collapsed spatial signal (A1) drops the address+distance double-count. What remains is precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. / over-merge — 38 clusters still fuse distinct co-located providers, because even one down-weighted spatial agreement can outvote a disagreeing name. The lever search (#625) — two negatives, then the first positive: a name/org/phone corroboration gate (A2/A3) — phone is an unreliable secondary identifier on NPPES (shared institutional switchboard lines), so it over-links and falsely corroborates co-phone distinct providers; and average-linkage clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. (A4) — the over-merged clusters are joined by STRONG shared-address edges, not weak bridges, so average-linkage can't split them and only trades away name-drift recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.. The over-merge is a scoring problem this data can't resolve, not a clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity.-topology one. The authorized-official discriminator is the first lever to beat the baseline — a reliable secondary identifier holds recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. so a higher threshold separates the co-located providers; a still-more-distinctive identifier (taxonomy / license) or a learned scorer over the FS featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. vector (#603) goes further. Config dominates the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (the pre-registered finding) — tracked as #625 / the selective-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' work (#602 / #603).
NPI-as-truth is conservative: a cluster fusing two NPIs is a candidate "same entity, two NPIs" surfaced for review, not an adjudicated error; an NPI split across distant addresses is geo-first behaving correctly, counted here as a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. miss. We resolve and report; interpretation is the consumer's.