NPPES NPI dedup benchmark (#617)
Generated by scripts/record-matcher/nppes-dedup-benchmark.ts. Sample: 300 TX NPIs with ≥1 alternate name → 816 records (2.7/NPI) from real registry + other-names + mailing-vs-practice variation. Matcher run BLIND to the NPI, EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-trained (labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free); geocoded 100.0% of addresses. The NPI is held-out ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy..
Headline — org-name truth (the honest grain): the shipped matcher resolves these records at F1 77.2% against org-name entity-truth — not the NPI-level 72.2%, which mostly measures NPI over-segmentation (one organization holds many subpart NPIs, so correct co-located merges are scored as errors). Same clusters, three rulers: NPI 72.2% → site 70.6% → org-name 77.2% (300 NPI → 443 site → 288 org-name classes); the climb is the yardstick getting honest, not the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' changing — gold-set validated (120/120 hard pairs = same org). Details in "Three truth grains" below.
The comparison-model levers (each toggled on, at the default threshold)
| modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 | ARI | over-merged |
|---|---|---|---|---|---|---|
| baseline (legacy: address-key + distance, levers OFF) | 48.3% | 76.5% | 59.2% | — | 0.591 | 39 |
| + inverse-address-frequency (#617, corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide) | 48.3% | 76.5% | 59.2% | +0.0pp | 0.591 | 39 |
| + collapsed spatial signal (A1, #625) | 52.2% | 76.5% | 62.1% | +2.9pp | 0.620 | 35 |
| + authorized-official discriminator (#625) | 51.1% | 76.5% | 61.3% | -0.8pp | 0.612 | 37 |
| + require name/org corroboration (A2, #625) | 51.8% | 76.5% | 61.7% | +0.5pp | 0.616 | 37 |
| + phone comparison (A3, #625) | 51.8% | 76.5% | 61.7% | +0.0pp | 0.616 | 37 |
| + average-linkage clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. (A4, #625, full A1–A4 stack) | 54.4% | 73.5% | 62.5% | +0.8pp | 0.624 | 38 |
| + taxonomy code-set discriminator (A5, #625) | 49.8% | 76.5% | 60.3% | -2.2pp | 0.602 | 36 |
Inverse-frequency weighting uses the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (3,317,267 distinct addresses over 9,260,504 providers) to down-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. a crowded shared address; collapsing the redundant address-key + distance comparisons into one spatial signal (A1) removes the double-count that let a shared address over-vote a disagreeing name. The address-frequency + A1 baseline is F1 62.1% at the default threshold; the authorized-official discriminator is roughly neutral there (-1.7pp) but enables a higher cutoff — it holds recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. where the baseline alone collapses, reaching 62.9% at threshold 16 (below), the first config past the baseline (#625).
Out-of-the-box (zero-config resolveEntities(records), levers default-on #86): F1 43.6% on this sample — essentially the baseline. The default auto-computes the address-frequency table over the INPUT records, and this benchmark deliberately sub-samples 816 records, so that table is too sparse to carry the inverse-frequency signal (a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. statistic you can't synthesize from a slice). It's not a regression (≥ baseline, no over-merge added) — it's the honest floor when the input isn't corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-scale. Fed the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table (the + inverse-address-frequency row above, what the CLI builds from the full source files) the SAME default reaches the 62.1% baseline. On a full-dataset dedup the input IS the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., so zero-config reaches the baseline on its own.
With all levers on, across the link threshold (the secondary lever)
| link threshold (bits) | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ARI | clusters | over-merged |
|---|---|---|---|---|---|---|
| 0 (default) | 49.8% | 76.5% | 60.3% | 0.602 | 298 | 36 |
| 4 | 51.5% | 76.5% | 61.5% | 0.614 | 302 | 36 |
| 8 | 52.8% | 76.5% | 62.5% | 0.624 | 306 | 35 |
| 12 | 52.8% | 76.5% | 62.5% | 0.624 | 306 | 35 |
| 16 | 53.4% | 76.5% | 62.9% ⭐ | 0.628 | 309 | 33 |
| 20 | 53.4% | 76.5% | 62.9% | 0.628 | 309 | 33 |
Best F1 62.9% (ARI 0.628) at threshold 16, vs 60.3% at the default — the threshold knob moves it 3pp. Higher thresholds trade recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. for precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1..
Shape + where the errors are (at the default threshold)
- records: 816 · true entities (NPIs): 300 · recovered clusters: 298
- candidate pairs blocked: 6867
- Over-merge (precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.): 36 clusters fuse ≥2 distinct NPIs (largest fuses 13; 212 records) — co-located providers sharing a clinic / billing address, linked by address agreement despite different names.
- Under-merge (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.): 81/300 NPIs split across >1 cluster — records at distant addresses (mailing vs practice) or with strong name drift the score didn't bridge.
Three truth grains — NPI → site → org-name (the over-segmentation correction)
NPI-as-truth OVER-SEGMENTSsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context.: one org holds many NPIs, so the matcher's correct co-located merges are scored as errors. Three yardsticks, same clusters: NPI (one entity per registration), site (subpart-flagged collapse + split an NPI's distinct addresses), and org-name — the gold-set-validated truth: same NPI ⇒ same entity (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. preserved) PLUS collapse co-located NPIs whose primary org names match (Jaccard ≥ 0.7), with NO reliance on the subpart flag (the gold set showed it misses 37%). The same 816 records carry 300 NPI → 443 site → 288 org-name classes. The org-name number is the honest one: it stops charging the matcher for the same-org merges the gold set proved are correct.
| config | truth | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 vs NPI | ARI | over-merged |
|---|---|---|---|---|---|---|---|
| FS full stack | NPI | 49.8% | 76.5% | 60.3% | — | 0.602 | 36 |
| FS full stack | site | 37.9% | 100.0% | 55.0% | -5.4pp | 0.549 | 80 |
| FS full stack | org-name | 57.3% | 78.4% | 66.2% | +5.9pp | 0.661 | 30 |
| FS full stack | org-name (coord) | 58.1% | 78.2% | 66.7% | +6.3pp | 0.666 | 28 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | NPI | 72.0% | 72.4% | 72.2% | — | 0.721 | 18 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | site | 55.7% | 96.3% | 70.6% | -1.6pp | 0.705 | 61 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | org-name | 81.7% | 73.1% | 77.2% | +5.0pp | 0.771 | 14 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | org-name (coord) | 82.2% | 72.5% | 77.0% | +4.8pp | 0.770 | 13 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (shipped default) | org-name (H3H3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id. res 11) | 82.2% | 72.5% | 77.0% | +4.8pp | 0.770 | 13 |
The F1 climbs as the yardstick gets honest: GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. 72.2% (NPI) → 70.6% (site) → 77.2% (org-name), the over-merge collapsing (18 → 14) and precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. rising (72.0% → 81.7%). The clusters are IDENTICAL across the three columns — the climb is purely the ruler ceasing to charge the matcher for the correct same-org merges the gold set proved (2026-06-16-dedup-gold-set-tx120.md: 120/120 same org, 0 genuine over-merges).
TiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. 2D — tightening the org-name ruler with the geocode coordinate. The org-name truth above blocks by the address STRING (addressFrequencyKey), so two records at one building whose text differs — 1504 Taub LOOP vs 1504 Taub LP STE 100 — key apart and the merge is still charged as an error. Block by the GEOCODED BUILDING instead (union co-located NPIs within 50 m whose org names agree, same Jaccard gate) and the truth tightens further: GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. org-name 77.2% → org-name-coord 77.0% (-0.1pp), 288 → 286 classes, over 300/300 geocoded NPIs. The string org-name F1 is a conservative LOWER bound; the coordinate one is tighter (the Jaccard gate still blocks distinct co-located orgs). Both are honest — the coordinate is the geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' key.
Task 4 (#109) — H3H3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id.-cell robustness check. Re-keying the same building-grain co-location on an H3 cellH3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id. (res 11) instead of the 50 m haversine radius — the deterministic, O(n) "geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' cell" the issue suggested — gives GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. org-name-h3H3Uber's hexagonal hierarchical geospatial indexing system. Mailwoman uses H3 cells at resolution 9 (~0.03 km²) for geo-first blocking in the matcher and for stable address primary keys in @mailwoman/address-id. 77.0% (286 classes), vs the haversine 77.0% (286). Within noise of the haversine grain — the ~77.0% coord-grain F1 is robust to the co-location method, not an artifact of the 50 m threshold.
Takeaways: (1) The dedup modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s REAL quality is the org-name F1 ~77.2%, not the NPI-level 72.2% — the difference was NPI over-segmentation, not modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' error. (2) The #625 lever was the YARDSTICK, not the scorer: the corroboration-featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. experiment (reverted) couldn't move precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. because the over-merge was mostly correct. (3) The remaining org-name over-merge (14 clusters) is the genuine frontier — small, approaching the ceiling's ~1.6% irreducible (2026-06-16-dedup-ceiling.md). (4) Trust the large evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.: a 50-NPI smoke misled on the site delta (+5–7pp vs this run's ±2pp).
Caveats: the programmatic site-truth is CONSERVATIVE (collapses only NPPES-flagged subparts; unflagged same-org pairs still read as over-merge), so it understatesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. the subpart correction. A hand-adjudicated gold set on the ambiguous co-located + two-site slice is the next refinement (#625).
Reading
The geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' foundation works: 100.0% of addresses placed, blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. + clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. clean — the geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. (the PeliasPeliasAn open-source geocoder, Mailwoman's spiritual predecessor./Nominatim-can't-do-this part) is not the bottleneck. The comparison-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' levers reach the address-frequency + A1 baseline at F1 62.1% (+2.9pp over baseline): inverse-frequency weighting restores full weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. to a rare shared address (stitching a provider's name-drifted records together — mostly recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.) while down-weighting a crowded one, and the collapsed spatial signal (A1) drops the address+distance double-count. What remains is precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. / over-merge — 36 clusters still fuse distinct co-located providers, because even one down-weighted spatial agreement can outvote a disagreeing name. The lever search (#625) — two negatives, then the first positive: a name/org/phone corroboration gate (A2/A3) — phone is an unreliable secondary identifier on NPPES (shared institutional switchboard lines), so it over-links and falsely corroborates co-phone distinct providers; and average-linkage clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. (A4) — the over-merged clusters are joined by STRONG shared-address edges, not weak bridges, so average-linkage can't split them and only trades away name-drift recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.. The over-merge is a scoring problem this data can't resolve, not a clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity.-topology one. The authorized-official discriminator is the first lever to beat the baseline — a reliable secondary identifier holds recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. so a higher threshold separates the co-located providers; a still-more-distinctive identifier (taxonomy / license) or a learned scorer over the FS featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. vector (#603) goes further. Config dominates the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (the pre-registered finding) — tracked as #625 / the selective-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' work (#602 / #603).
NPI-as-truth is conservative: a cluster fusing two NPIs is a candidate "same entity, two NPIs" surfaced for review, not an adjudicated error; an NPI split across genuinely-distant addresses is geo-first behaving correctly, counted here as a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. miss. We resolve and report; interpretation is the consumer's.