NPPES NPI dedup benchmark (#617)
Generated by scripts/record-matcher/nppes-dedup-benchmark.ts. Sample: 300 TX NPIs with ≥1 alternate name → 816 records (2.7/NPI) from real registry + other-names + mailing-vs-practice variation. Matcher run BLIND to the NPI, EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-trained (labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free); geocoded 100.0% of addresses. The NPI is held-out ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy..
Headline — org-name truth (the honest grain): the shipped matcher resolves these records at F1 70.1% against org-name entity-truth — not the NPI-level 65.2%, which mostly measures NPI over-segmentation (one organization holds many subpart NPIs, so correct co-located merges are scored as errors). Same clusters, three rulers: NPI 65.2% → site 60.9% → org-name 70.1% (300 NPI → 443 site → 288 org-name classes); the climb is the yardstick getting honest, not the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' changing — gold-set validated (120/120 hard pairs = same org). Details in "Three truth grains" below.
The comparison-model levers (each toggled on, at the default threshold)
| modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 | ARI | over-merged |
|---|---|---|---|---|---|---|
| baseline (legacy: address-key + distance, levers OFF) | 45.5% | 42.3% | 43.9% | — | 0.437 | 38 |
| + inverse-address-frequency (#617, corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide) | 52.1% | 77.6% | 62.4% | +18.5pp | 0.623 | 36 |
| + collapsed spatial signal (A1, #625) | 52.2% | 77.6% | 62.4% | +0.1pp | 0.623 | 36 |
| + authorized-official discriminator (#625) | 50.5% | 77.6% | 61.2% | -1.3pp | 0.610 | 38 |
| + require name/org corroboration (A2, #625) | 51.4% | 77.6% | 61.8% | +0.7pp | 0.617 | 38 |
| + phone comparison (A3, #625) | 51.4% | 77.6% | 61.8% | +0.0pp | 0.617 | 38 |
| + average-linkage clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. (A4, #625, full A1–A4 stack) | 54.6% | 74.7% | 63.1% | +1.3pp | 0.630 | 37 |
Inverse-frequency weighting uses the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (3,317,267 distinct addresses over 9,260,504 providers) to down-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. a crowded shared address; collapsing the redundant address-key + distance comparisons into one spatial signal (A1) removes the double-count that let a shared address over-vote a disagreeing name. The address-frequency + A1 baseline is F1 62.4% at the default threshold; the authorized-official discriminator is roughly neutral there (+0.6pp) but enables a higher cutoff — it holds recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. where the baseline alone collapses, reaching 64.8% at threshold 4 (below), the first config past the baseline (#625).
Out-of-the-box (zero-config resolveEntities(records), levers default-on #86): F1 44.3% on this sample — essentially the baseline. The default auto-computes the address-frequency table over the INPUT records, and this benchmark deliberately sub-samples 816 records, so that table is too sparse to carry the inverse-frequency signal (a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. statistic you can't synthesize from a slice). It's not a regression (≥ baseline, no over-merge added) — it's the honest floor when the input isn't corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-scale. Fed the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (the + inverse-address-frequency row above, what the CLI builds from the full source files) the SAME default reaches the 62.4% baseline. On a full-dataset dedup the input IS the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., so zero-config reaches the baseline on its own.
With all levers on, across the link threshold (the secondary lever)
| link threshold (bits) | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ARI | clusters | over-merged |
|---|---|---|---|---|---|---|
| 0 (default) | 54.6% | 74.7% | 63.1% | 0.630 | 314 | 37 |
| 4 | 57.7% | 74.0% | 64.8% ⭐ | 0.647 | 321 | 36 |
| 8 | 55.6% | 62.8% | 59.0% | 0.589 | 357 | 37 |
| 12 | 56.4% | 60.2% | 58.3% | 0.582 | 365 | 36 |
| 16 | 53.6% | 38.2% | 44.6% | 0.445 | 501 | 34 |
| 20 | 52.9% | 31.5% | 39.5% | 0.393 | 544 | 34 |
Best F1 64.8% (ARI 0.647) at threshold 4, vs 63.1% at the default — the threshold knob moves it 2pp. Higher thresholds trade recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. for precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1..
Shape + where the errors are (at the default threshold)
- records: 816 · true entities (NPIs): 300 · recovered clusters: 314
- candidate pairs blocked: 8282
- Over-merge (precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.): 37 clusters fuse ≥2 distinct NPIs (largest fuses 13; 182 records) — co-located providers sharing a clinic / billing address, linked by address agreement despite different names.
- Under-merge (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.): 90/300 NPIs split across >1 cluster — records at distant addresses (mailing vs practice) or with strong name drift the score didn't bridge.
Three truth grains — NPI → site → org-name (the over-segmentation correction)
NPI-as-truth OVER-SEGMENTSsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context.: one org holds many NPIs, so the matcher's correct co-located merges are scored as errors. Three yardsticks, same clusters: NPI (one entity per registration), site (subpart-flagged collapse + split an NPI's distinct addresses), and org-name — the gold-set-validated truth: same NPI ⇒ same entity (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. preserved) PLUS collapse co-located NPIs whose primary org names match (Jaccard ≥ 0.7), with NO reliance on the subpart flag (the gold set showed it misses 37%). The same 816 records carry 300 NPI → 443 site → 288 org-name classes. The org-name number is the honest one: it stops charging the matcher for the same-org merges the gold set proved are correct.
| config | truth | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 vs NPI | ARI | over-merged |
|---|---|---|---|---|---|---|---|
| FS full stack | NPI | 54.6% | 74.7% | 63.1% | — | 0.630 | 37 |
| FS full stack | site | 40.9% | 96.3% | 57.4% | -5.6pp | 0.574 | 85 |
| FS full stack | org-name | 62.8% | 76.5% | 69.0% | +5.9pp | 0.689 | 29 |
| FS full stack | org-name (coord) | 69.5% | 75.2% | 72.3% | +9.2pp | 0.722 | 23 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | NPI | 58.4% | 73.7% | 65.2% | — | 0.651 | 27 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | site | 44.5% | 96.5% | 60.9% | -4.2pp | 0.609 | 74 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | org-name | 66.2% | 74.4% | 70.1% | +4.9pp | 0.700 | 21 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | org-name (coord) | 75.1% | 74.8% | 74.9% | +9.8pp | 0.749 | 15 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | org-name (H3H3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id. res 11) | 75.1% | 74.8% | 74.9% | +9.8pp | 0.749 | 15 |
The F1 climbs as the yardstick gets honest: GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. 65.2% (NPI) → 60.9% (site) → 70.1% (org-name), the over-merge collapsing (27 → 21) and precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. rising (58.4% → 66.2%). The clusters are IDENTICAL across the three columns — the climb is purely the ruler ceasing to charge the matcher for the correct same-org merges the gold set proved (2026-06-16-dedup-gold-set-tx120.md: 120/120 same org, 0 genuine over-merges).
TiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. 2D — tightening the org-name ruler with the geocode coordinate. The org-name truth above blocks by the address STRING (addressFrequencyKey), so two records at one building whose text differs — 1504 Taub LOOP vs 1504 Taub LP STE 100 — key apart and the merge is still charged as an error. Block by the GEOCODED BUILDING instead (union co-located NPIs within 50 m whose org names agree, same Jaccard gate) and the truth tightens further: GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. org-name 70.1% → org-name-coord 74.9% (+4.9pp), 288 → 278 classes, over 300/300 geocoded NPIs. The string org-name F1 is a conservative LOWER bound; the coordinate one is tighter (the Jaccard gate still blocks distinct co-located orgs). Both are honest — the coordinate is the geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' key.
Task 4 (#109) — H3H3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id.-cell robustness check. Re-keying the same building-grain co-location on an H3 cellH3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id. (res 11) instead of the 50 m haversine radius — the deterministic, O(n) "geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' cell" the issue suggested — gives GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. org-name-h3H3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id. 74.9% (278 classes), vs the haversine 74.9% (278). Within noise of the haversine grain — the ~74.9% coord-grain F1 is robust to the co-location method, not an artifact of the 50 m threshold.
Takeaways: (1) The dedup modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s REAL quality is the org-name F1 ~70.1%, not the NPI-level 65.2% — the difference was NPI over-segmentation, not modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' error. (2) The #625 lever was the YARDSTICK, not the scorer: the corroboration-featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. experiment (reverted) couldn't move precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. because the over-merge was mostly correct. (3) The remaining org-name over-merge (21 clusters) is the genuine frontier — small, approaching the ceiling's ~1.6% irreducible (2026-06-16-dedup-ceiling.md). (4) Trust the large evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.: a 50-NPI smoke misled on the site delta (+5–7pp vs this run's ±2pp).
Caveats: the programmatic site-truth is CONSERVATIVE (collapses only NPPES-flagged subparts; unflagged same-org pairs still read as over-merge), so it understatesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. the subpart correction. A hand-adjudicated gold set on the ambiguous co-located + two-site slice is the next refinement (#625).
Reading
The geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' foundation works: 100.0% of addresses placed, blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. + clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. clean — the geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. (the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor./Nominatim-can't-do-this part) is not the bottleneck. The comparison-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' levers reach the address-frequency + A1 baseline at F1 62.4% (+18.6pp over baseline): inverse-frequency weighting restores full weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. to a rare shared address (stitching a provider's name-drifted records together — mostly recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.) while down-weighting a crowded one, and the collapsed spatial signal (A1) drops the address+distance double-count. What remains is precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. / over-merge — 37 clusters still fuse distinct co-located providers, because even one down-weighted spatial agreement can outvote a disagreeing name. The lever search (#625) — two negatives, then the first positive: a name/org/phone corroboration gate (A2/A3) — phone is an unreliable secondary identifier on NPPES (shared institutional switchboard lines), so it over-links and falsely corroborates co-phone distinct providers; and average-linkage clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. (A4) — the over-merged clusters are joined by STRONG shared-address edges, not weak bridges, so average-linkage can't split them and only trades away name-drift recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.. The over-merge is a scoring problem this data can't resolve, not a clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity.-topology one. The authorized-official discriminator is the first lever to beat the baseline — a reliable secondary identifier holds recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. so a higher threshold separates the co-located providers; a still-more-distinctive identifier (taxonomy / license) or a learned scorer over the FS featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. vector (#603) goes further. Config dominates the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (the pre-registered finding) — tracked as #625 / the selective-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' work (#602 / #603).
NPI-as-truth is conservative: a cluster fusing two NPIs is a candidate "same entity, two NPIs" surfaced for review, not an adjudicated error; an NPI split across genuinely-distant addresses is geo-first behaving correctly, counted here as a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. miss. We resolve and report; interpretation is the consumer's.