Learned-scorer CROSS-STATE generalization (#603 Tier 2) — train TX, evaluate NY
Generated by scripts/record-matcher/learned-scorer-crossstate-eval.ts. The GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. + LR are trained on 2000 TX NPIs (5641 records) and used to cluster 2000 held-out NY NPIs (5303 records) — a stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' never saw — through the same resolveEntities pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. (FS baseline / GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. scorer / LR scorer), best F1 over a fine per-scorer threshold sweep. This is the generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure. axis the within-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. held-out-NPI A/B couldn't cover.
Result — NY clustering F1 (GBT/LR trained on TX)
| scorer | precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. | recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. | F1 | ΔF1 vs FS | over-merged |
|---|---|---|---|---|---|
| FS baseline (EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.-fit) | 12.2% | 38.6% | 18.6% | — | 346 |
| logistic regression | 12.5% | 75.4% | 21.5% | +2.9pp | 261 |
| GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (fresh TX fit) | 61.6% | 27.7% | 38.2% | +19.6pp | 50 |
| SHIPPED bundled modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (default-on candidate) | 74.6% | 27.8% | 40.5% | +21.9pp | 58 |
The bundled row is the actual shipped DEDUP_GBT_MODEL (the default-on candidate), evaluated on NY — a stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. it never trained on. The "fresh TX fit" row retrains per run for comparison.
The GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. win GENERALIZES across statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. — trained on TX, it still beats the FS baseline on NY clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. F1 (38.2% vs 18.6%, +19.6pp). The learned scorer isn't fitting TX-specific structure; the over-merge signal it learns transfers. This is the strongest evidence yet for the #603 production GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. — one modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', trained once, helps a stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. it never saw.
Caveats
A single train/evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. pair (TX→NY), one geocoded sample each, a compact pure-Node GBTGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. (120 rounds, depth 3). The FS arm is the benchmark baseline (same modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'), so the comparison is fair. Absolute F1 differs from the within-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. A/B because the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. population + over-merge density differ by stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.. NPI-as-truth is conservative. The within-stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. held-out-NPI A/B (learned-scorer-clustering-eval.ts) is the companion; together they bound the generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure. question a production GBMGBT (Gradient Boosted Trees). A non-linear machine learning model that combines many weak decision trees into a strong predictor. Mailwoman uses a GBT as an optional learned scorer for single-dataset dedup, improving F1 by 5–7 percentage points over the Fellegi-Sunter baseline. must answer._