#625 — dedup ceiling: the irreducible over-merge of co-located providers
Generated by scripts/record-matcher/dedup-ceiling.ts. Full TX set — all 163,739 type-2 (org) providers (scanned the entire 9.6M-row registry; the cap 200000 was not binding). Geocode-free: "same address" = the matcher's addressFrequencyKey; "same name" = normalized tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. Jaccard over the legal business name (corporate suffixes + articles stripped, domain words kept). NPI is the distinctness truth (different NPI = different provider). τ = 0.7.
Trust the full-population number. Smaller "first-N in file order" samples badly OVERSTATE the collision rate — it fell 51% (cap 5k) → 14.5% (cap 50k) → 6.1% (full TX) as the sample de-biased, because NPPES is ~NPI-assignment order and the earliest NPIs are large multi-subpart institutions. The figures below are the full TX population.
Co-location prevalence
- 21970 addresses host ≥2 distinct NPIs (18.7% of 117296 addresses).
- 68413 providers sit at a shared address (41.8% of 163739).
- 212079 co-located distinct-NPI pairs (the over-merge population).
Name separability of co-located distinct providers
| org-name Jaccard | pairs | share | meaning |
|---|---|---|---|
| ≥ 0.7 (collision) | 12989 | 6.1% | ~identical names → irreducible over-merge |
| 0.3–0.7 | 13428 | 6.3% | partial — separable with a good modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' |
| < 0.3 | 185662 | 87.5% | clearly different names → separable |
Of the 12989 collision pairs, 8715 (67.1%) also share a phone — so phone (a shared institutional switchboard) does NOT separate them either; if anything it over-links. This is why the benchmark found phone an unreliable secondary identifier.
Splitting the collisions: NPI over-segmentation vs distinct providers
A collision (same address, ~same name, often same phone) with DIFFERENT NPIs is usually one organization holding multiple NPIs (subparts / departments) — where merging is correct and NPI-as-truth is over-segmenting, not a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' error. NPPES's own fields separate the two cases:
| collision pair is… | pairs | share of collisions | merging it is… |
|---|---|---|---|
| same authorized official | 8665 | 66.7% | correct — one org, many NPIs (NPI over-segmentssegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context.) |
| different official AND different specialty | 3382 | 26.0% | a genuine distinct co-located provider — true over-merge |
| (remainder: different official, same specialty) | 942 | 7.3% | ambiguous — needs adjudication |
The ceiling
The raw collision rate is 6.1% of co-located distinct-NPI pairs — but only 1.6% of co-located pairs are genuinely distinct providers indistinguishable by name (different official + specialty). Most collisions are NPI over-segmentation (66.7% share an authorized official), where a merge is correct and NPI-truth penalizes it wrongly.
This is the answer to "how good is good enough," and it has two parts:
- Measured against NPI-as-truth, F1 is capped well below 0.85 — but the cap is NPI-truth's own over-segmentation, not unseparable data. The matcher correctly fuses co-located same-org subpart-NPIs (the 6.1% name-collisions, 67% of which share an authorized official); NPI-truth scores those correct merges as over-merge errors. You can't reach 0.85 against this yardstick without UN-merging correct subparts — which would be wrong. The round 0.85 target is unreachable under NPI-truth and should be dropped.
- The real irreducible over-merge — distinct co-located providers with identical names — is only ~1.6% of the co-located population. Against an entity-level truth (subpart-aware), the achievable ceiling is much higher. But that ceiling can only be MEASURED with an entity-level / adjudicated gold set — NPI-truth alone can't tell a correct subpart-merge from a true over-merge. This is why the gold set (the "second comparison") is necessary, not optional.
Recommendation: drop 0.85. Set the bar against a subpart-aware / adjudicated entity truth, report NPI-level AND entity-level side by side, and target "separate the ~1.6% distinct co-located pairs the GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. can still reach" rather than a round F1. The GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline.'s corroboration-featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. work (#625 revised) attacks exactly that separable slice.
Caveats
- Geocode-free + exact address key. "Co-located" here is an exact normalized-address match; geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. would add near-but-not-exact neighbors (suite splits, slightly different formatting), which can only RAISE the collision count. So this is a LOWER bound on the irreducible over-merge.
- RecallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. side under-measured. NPPES same-NPI records almost always share an address or the org name, so the recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. floor looks ~1.0 here; real-world feeds with distant + name-drifted same-entity records would lower it. The F1 ceiling reported tracks the PRECISIONprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. constraint, which is the binding one for the over-merge problem.
- TokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-Jaccard ≠ the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s name comparison. A proxy for separability; the GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. uses the FS agreement levels. The collision SET (sim ≥ τ + shared phone) is robust to the exact similarity metric.