Cross-dataset correlation (#618 / #87 real-data run)
Generated by scripts/record-matcher/cross-dataset-correlation.ts. TX-scoped, ≤2000 rows per source geocoded, resolved BLIND across sources (geo-first block → Fellegi-SunterFellegi-SunterA probabilistic record linkage model that computes match probability from agreement-level log-likelihood ratios: log₂(m/u) where m is the probability of agreement given a true match and u is the probability of agreement by chance. Mailwoman learns m and u label-free via expectation-maximization. + EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. → cluster) with the proven levers default-on (#86). The sources share no key; an entity spanning ≥2 sources is a cross-dataset link we surface for review — interpretation is the consumer's.
Sources
| source | rows | what it is |
|---|---|---|
txhhsc-nursing | 927 | TX HHSC licensed nursing facilities |
fcc-rhc | 2000 | FCC Rural Health Care posted-services filings |
nppes | 2000 | NPPES organization NPIs |
fcc-rhc-commitments | 2000 | FCC RHC funding commitments (Filing + Participating HCP, exploded) |
Combined: 6927 records, geocoded 100.0%. Resolved to 3584 entities from 534804 candidate pairs.
Matched with the proven levers default-on (#86): collapsed spatial (A1) + inverse-address-frequency, fed a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-wide table built from the full source files (119,613 distinct addresses over 208,856 TX rows — a crowded shared campus is down-weighted as weak identity evidence).
Scored with the Fellegi-SunterFellegi-SunterA probabilistic record linkage model that computes match probability from agreement-level log-likelihood ratios: log₂(m/u) where m is the probability of agreement given a true match and u is the probability of agreement by chance. Mailwoman learns m and u label-free via expectation-maximization. baseline (learnedScorer: false): cross-dataset link discovery is recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.-oriented — the same facility under different operational names across sources is the signal — so the dedup-calibrated GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. default (#603), which is trained to REJECT "same place, different name," is pinned off here. A cross-objective GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. threshold is the follow-up (#655).
Cross-dataset links (entities spanning ≥2 sources)
219 entities resolve across ≥2 sources.
| source pair | entities linked |
|---|---|
| fcc-rhc ↔ fcc-rhc-commitments | 201 |
| fcc-rhc ↔ nppes | 16 |
| fcc-rhc-commitments ↔ nppes | 11 |
| fcc-rhc-commitments ↔ txhhsc-nursing | 5 |
| nppes ↔ txhhsc-nursing | 5 |
| fcc-rhc ↔ txhhsc-nursing | 1 |
Of those, 10 spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. all three sources.
Spot-check — the first 12 cross-source entities (verify by eye)
| entity | sources | name (representative) | coordinate |
|---|---|---|---|
| entity-939 | fcc-rhc, fcc-rhc-commitments, nppes | uvalde county hospital authority | 29.2154, -99.7782 |
| entity-1055 | fcc-rhc, fcc-rhc-commitments, nppes | hunt memorial hospital district | 33.1233, -96.1228 |
| entity-992 | fcc-rhc, fcc-rhc-commitments, nppes | medina county hospital district | 29.3533, -99.1621 |
| entity-943 | fcc-rhc, fcc-rhc-commitments, nppes | winkler county memorial hospital | 31.8496, -103.0903 |
| entity-1100 | fcc-rhc, fcc-rhc-commitments, nppes | oakbend medical center | 29.5780, -95.7705 |
| entity-945 | fcc-rhc, fcc-rhc-commitments, nppes | castro county hospital district | 34.5410, -102.3215 |
| entity-1076 | fcc-rhc, fcc-rhc-commitments, nppes | scott and white hospital brenham | 30.1454, -96.3992 |
| entity-977 | fcc-rhc, fcc-rhc-commitments, nppes | stonewall memorial hospital | 33.1405, -100.2253 |
| entity-1009 | fcc-rhc, fcc-rhc-commitments, nppes | yoakum community hospital | 29.2934, -97.1469 |
| entity-732 | fcc-rhc, fcc-rhc-commitments, txhhsc-nursing | schleicher county medical center | 30.8707, -100.5923 |
| entity-1075 | fcc-rhc, fcc-rhc-commitments | baylor scott and white health bswh | 32.7940, -96.7657 |
| entity-1071 | fcc-rhc, fcc-rhc-commitments | christus health | 29.4704, -98.6862 |
Reading
4 datasets with no shared key — a provider registry, a federal funding program (two of its forms, the commitments form exploded into its Filing + Participating HCP per row), and a stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. facility registry — resolve into a single entity modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' where 219 entities are corroborated by ≥2 independent sources (10 by all three kinds), purely on geocoded location + name/org agreement, in pure Node (no Elasticsearch, no server). Each cross-source entity is a candidate "same place, multiple records" surfaced for review; whether a correlation means anything is the data consumer's call, not ours.