Learned-scorer CLUSTERING A/B (#603 Tier 2) — does a learned scorer beat the FS baseline on the assembled output?

Generated by scripts/record-matcher/learned-scorer-clustering-eval.ts. 2000 TX NPIs → 5641 records, geocoded; split by NPI into ~3724 train / ~1917 evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. records over 4 seeds (the GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline./LR never see an evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. NPI's records). The held-out EVALevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. records are clustered three ways through the SAME resolveEntities pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. (block → score → connected-components): the FS baseline (address-frequency + collapsed-spatial, EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-fit), the GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. as the link scorer (the new ResolveConfig.scorer hook), and the LR. Best F1 over a fine per-scorer link-threshold sweep, averaged across seeds. This is the dedup benchmark's clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. metric — the definitive test the pairwise probe (#637/#640) deferred.

Result — eval clustering F1 (best over threshold, mean ± std over 4 seeds, ~1917 held-out records)

scorer	precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1.	recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.	F1	ΔF1 vs FS	over-merged clusters
FS baseline (EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-fit)	45.3%	71.3%	55.3% ± 3.2	—	94
logistic regression	54.2%	59.7%	56.7% ± 2.4	+1.4pp	79
gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.-boosted trees	61.2%	62.9%	60.5% ± 2.7	+5.2pp	69

ΔF1 (GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. − FS): +5.2pp mean, GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. > FS in 4/4 seeds.

The learned scorer beats the FS baseline on the assembled clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. output — GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. F1 60.5% vs FS 55.3% (+5.2pp mean, 4/4 seeds), driven by a large PRECISIONprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. gain that cuts the over-merge — the #625 problem. The pairwise gain (#640) DOES translate to the entity-resolution metric. This confirms the #603 GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. as a real dedup lever and justifies the production build (offline XGBoost/LightGBM → tree JSON, the scorer hook for inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.). The honest next axis is cross-STATEregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure. (train-TX / evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-other-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.) and a tuned GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. on more featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them..

Per-seed F1

seed	evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. records	FS	LR	GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline.
1	1957	50.4%	54.3%	57.1%
2	2000	55.0%	54.7%	59.8%
3	1841	59.4%	60.4%	64.6%
4	1870	56.2%	57.4%	60.4%

Caveats

In-domain (TX), a held-out-NPI split (NOT a held-out STATEregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. — cross-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure. is the next axis, the #603 train-TX/evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-other-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. design). The FS arm IS the benchmark baseline (same modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'), so the comparison is fair. The GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. is a compact pure-Node implementation (120 rounds, depth 3), not a tuned XGBoost/LightGBM — a real GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. with more NPIs/featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. could move the number further. Thresholds are swept per scorer (FS in bits, learned scorers in logitslogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities.), each at its own best operating point — note a 300-NPI smoke MISLED (FS ahead): too few co-located collisions to exhibit the over-merge, which only bites at scale, so trust the larger evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.. NPI-as-truth is conservative (a cross-NPI merge is a candidate, not necessarily an error)._