Learned-scorer probe (#603) — does a model beat Fellegi-Sunter on the FS feature vector?

Generated by scripts/record-matcher/learned-scorer-eval.ts. 1500 TX NPIs → 4182 records, geocoded. Candidate pairs are split BY NPI into train/test (no NPI's records cross the split), repeated over 8 seeds to bound split variance. Two learned scorers over the FS agreement pattern + over-merge interaction featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. (spatial-exact × name-disagree, spatial-exact × org-disagree, address crowdedness) — featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. FS structurally cannot express — vs the EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-fitted FS scorer, on the held-out test pairs: an L2 logistic regression (linear) and gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-boosted trees (non-linear, the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' #603 names). AUC is threshold-free (does it RANK matches above non-matches?). The FS scorer is fit unsupervised on ALL pairs, so the comparison slightly favors FS — it has already seen the test pairs (labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free), the learned scorers have not.

Result — mean over 8 NPI-splits (~17432 test pairs/split, ~1026 matches)

scorer	ROC-AUC (mean±std)	ΔAUC vs FS	best F1 (mean)
Fellegi-SunterFellegi-SunterA probabilistic record linkage model that computes match probability from agreement-level log-likelihood ratios: log₂(m/u) where m is the probability of agreement given a true match and u is the probability of agreement by chance. Mailwoman learns m and u label-free via expectation-maximization. (EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-fit)	0.9420 ± 0.0053	—	72.6%
logistic regression (linear)	0.9477 ± 0.0035	+0.0057	76.9%
gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-boosted trees	0.9597 ± 0.0015	+0.0177	79.1%

ΔAUC (LR − FS): +0.0057 ± 0.0030, LR > FS in 8/8 seeds.

Robustness: the ΔAUC is small but consistent — std 0.0030 across seeds, SE ±0.0011 → ≈5.4σ above zero, 8/8 seeds in LR's favour. At the operating point the gap is larger: ΔF1 +4.3pp (72.6% → 76.9%), because the interaction featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. sharpen the hard co-located band near the decision boundary even where overall ranking barely moves.

The tree extends the linear gain — GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. beats the LR by ΔAUC +0.0121 (8/8 seeds), +0.0177 over FS, ΔF1 +6.6pp. Non-linear interactions the hand-crafted featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. miss carry additional signal — a real GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (XGBoost/LightGBM, more NPIs, more featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them.) is worth building.

Per-seed

seed	test pairs	FS AUC	LR AUC	GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. AUC
1	19340	0.9311	0.9415	0.9599
2	18520	0.9409	0.9442	0.9572
3	16156	0.9499	0.9527	0.9624
4	17450	0.9401	0.9509	0.9607
5	17105	0.9422	0.9469	0.9599
6	17416	0.9476	0.9507	0.9606
7	16231	0.9437	0.9479	0.9590
8	17236	0.9406	0.9466	0.9580

The LR beats FS consistently — it wins 8/8 seeds and lifts the operating-point F1 by +4.3pp (72.6% → 76.9%). The ΔAUC is small (+0.0057) only because FS already ranks well (0.9420); the gain concentrates at the decision boundary, exactly where the interaction featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. (which FS structurally can't express) bite. This greenlights the #603 learned scorer: a GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. — non-linear over the same featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. — is the principled generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure. of the hand-tuned #625 levers and should extend this linear gain. Honest framing: the linear headroom is modest, so the GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline.'s job is to widen a real-but-small margin, not to unlock a step change past the 64.7% dedup plateau on its own — the reliable secondary identifier (#625) is still the larger lever.

Caveats

In-domain (TX only), 1500 NPIs, PAIRWISE ranking (not the assembled clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. metric the dedup benchmark reports against the 64.7% baseline — a better pairwise scorer need not translate 1:1 to cluster F1). The GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. is a compact pure-Node implementation (120 boosting rounds, depth 3), a faithful stand-in for an offline XGBoost/LightGBM but not tuned. The split is by NPI so there's no record-level leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits., but the address-frequency featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. is a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. statistic over all NPIs (a population prior, not per-pair leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits.), and the FS scorer is EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-fit on all pairs including the test subset (standard for labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free FS — it makes the learned scorers' win the harder result). At ~17432 test pairs/split both AUC and F1 are stable across seeds (per-seed table). The definitive test remains a GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. A/B on the clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. metric with a train-TX / evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-held-out-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. split (#603 TiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. 2)._