Record-matcher session handoff — 2026-06-14 (Heather's shift)

For the next Claude. You have no memory of this session beyond the auto-loaded project-record-matcher.md memory file; this is the full narrative + file map so you can pick up cleanly. Everything below is on branch feat/record-matcher-foundation, draft PR #607, Project #5.

TL;DR

We revived the contact/organization record-matcher mailwoman was originally built to serve (clinic federal/stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. funding eligibility: normalizenormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input. → geocode → match → QGIS → leads; it imploded years ago on the record-matching step). In one shift we built the entire geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' entity-resolution system end to end, labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free, across four new workspaces and 14 commits, with 121 passing tests in the new packages (+170 corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. tests still green through a migration). Nothing is merged; it's all a draft PR awaiting review/CI/split. The only thing left to run it on real data is a CLI command that injects the heavy geocoder (operator-verifiable; the weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values./shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. aren't in the worktree).

What got built — the cascade

CSV / SQLite -ingest-> normalize (name, org, address) -[GeocodeAddress seam]-> SourceRecord
   -> @mailwoman/match  BLOCK (geo | canonical | phone | email)
                        SCORE (Jaro-Winkler, distance buckets, Fellegi-Sunter;
                               m/u learned label-free by EM; rare values up-weighted by TF)
                        CLUSTER (connected components)
   -> @mailwoman/registry  ResolvedEntity[] -> toGeoJSON -> QGIS

The thesis, proven in a capstone test: two records reading 123 main st and 123 main street apt 2 — different strings — resolve to one entity because they share a location and a name. BlockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. is geographic, not textual. That's why this version works where the string-first v0 imploded.

The four new workspaces (and what's in each file)

• @mailwoman/formatter (#599) — the inverse of the parser.

  • format.ts: formatAddress(components, country, opts) (ComponentTagcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. to idiomatic string, wraps @fragaria/address-formatter), formatFromClassificationMap (bridges the legacy rule-classifier vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it.), reconcileComponents, toOpenCageComponents.
  • key.ts: canonicalKey(components) + normalizeAddressToken — the normalized, deterministic match keycanonical keyA deterministic, normalized string representation of an address produced by @mailwoman/formatter. Lowercase, abbreviation-expanded, punctuation-stripped — so '123 Main St' and '123 MAIN STREET' produce the same key. Used for blocking in the matcher. blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. collides on.
  • corpus/src/format.ts is now a thin re-export of this (~170 dup lines removed, @fragaria dep dropped from corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.). core/formatter was intentionally NOT migrated — formatter type-deps core's ComponentTag, so core depending on formatter would be a forbidden tsc -b reference cycle. Don't try it; the fix needs the schema types in a shared base (a separate refactor).

• @mailwoman/record (#600) — the canonicalize layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6.. Plain TypeScript interfaces only (operator decision: no Nexus/TypeORM/JSON-schema machinery; use Kysely if a DB is ever needed).

  • address.ts: PostalAddress (components + canonicalKey + optional AddressGeocode
    • formatted), AddressGeocode (mirrors mailwoman GeocodeResult: ResolutionTier + uncertaintyMeters + hierarchy + poBox/multiUnit flags), toPostalAddress, withGeocode.
  • name.ts: parsePersonName — rule-based positional parser (the python-nameparser recipe), comma-inversion, leading titles, trailing suffixes, surname particle stored separately (de la Vega to particle de la + family Vega), nickname extraction. Western/romanized only; does NOT map nicknames to roots (lossy/gendered — that belongs in the matcher as a fuzzy agreement level).
  • organization.ts: canonicalizeOrganizationName — Winkler designation-strip (Acme Corp equals Acme Corporation, LLC), DBA split, ampersand to "and", intra-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. period/apostrophe removal (S.A. to sa), leading-The drop. ISO 20275 ELF + cleanco designation list.

• @mailwoman/match (#601) — the matcher: block, score, cluster. Generic over the record shape R (no dep on record).

  • comparators.ts: jaro, jaroWinkler (verified vs canonical refs), nameSimilarity (compound-surname edit/LCS fallback for the J-W blind spot), levenshteinSimilarity.
  • fellegi-sunter.ts: ComparisonLevel (m, u, minSimilarity, maxKm), Comparison<R>, similarityComparison, levelWeight (log2 m/u), priorWeight, probabilityFromWeight (overflow-safe), scorePair (returns { weight, probability, contributions }), decide (link/review/non-link). Also the TermFrequencyAdjustment hook.
  • em.ts: estimateParameters (Winkler EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. — labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free m/u + lambda fitting), agreementPattern. The keystone: trains with NO ground truthground truthThe correct answer for an example, used as the standard a prediction is graded against. Mailwoman's ground truth is the hand-labeled golden set; its quality caps achievable accuracy..
  • tf.ts: buildTermFrequencyTable (on-the-fly relative freqs — no Census table), withTermFrequency (rare-value agreement counts more; Vijayan beats Smith).
  • blocking.ts: geoCellKey (generous neighbour-expanded lat/lon grid — the geo-first primary block), exactKey, conjunction, block (union of keys, deduped pairs, oversized blocks reported not silently dropped). BlockingKey<R>, LatLon.
  • distance.ts: haversineKm, distanceComparison (Splink DistanceInKMAtThresholds — bucket distance into FS levels), DEFAULT_DISTANCE_LEVELS.
  • clustering.ts: cluster (connected-components union-find; pairwise scores are non-transitive so this is a required distinct stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).; threshold = precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1./recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. knob; over-merge caveat — centroid-linkage is the documented refinement), representative (most-complete record).
  • pipeline.test.ts: the block-score-cluster capstone.

• @mailwoman/registry (#604) — the application. Depends on match + record.

  • types.ts: SourceRecord (the messy row, normalized), ResolvedEntity, minimal GeoJSON types.
  • resolve.ts: resolveEntities(records, config) runs block-score-cluster with geocode-firstgeocode-firstThe matcher's core design principle: resolve addresses to geographic coordinates first, then compare the resolved places — not the raw address strings. Two records at the same coordinates match even if one says '123 Main St' and the other says '123 MAIN STREET.' defaults (buildDefaultModel: name+org+address-key+distance comparisons; defaultBlockingKeys: geo+canonical+phone+email) + optional trainEM.
  • geojson.ts: toGeoJSON(entities) to a QGIS Point FeatureCollection (recordCount, cohesion, name/org/address, geocode tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. as properties).
  • ingest.ts: parseCsv, ColumnMapping, ingestRows (pure column-map + normalizenormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input.), GeocodeAddress (the injected geocoder seam), geocodeAddressVia (the adapter that wires mailwoman's real parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.+geocode into the seam; RawGeocode is structurally a GeocodeResult subset, so this package never imports the heavy runtime).

Locked decisions (don't re-litigate)

1. Home: new workspaces in sister-software/mailwoman (not isp-nexus, not a fresh repo). isp-nexus holds the legacy "bones" we ported from (isp-nexus/universe/mailwoman/contacts, /organization, /postal).
2. Matcher v1: classical Fellegi-SunterFellegi-SunterA probabilistic record linkage model that computes match probability from agreement-level log-likelihood ratios: log₂(m/u) where m is the probability of agreement given a true match and u is the probability of agreement by chance. Mailwoman learns m and u label-free via expectation-maximization. core + EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. (labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag.-free), with modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' as selective additions — NOT modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'-first, NOT heuristics-only. Evidence-backed.
3. Schema: plain TS interfaces, no ORM/JSON-schema generation. Kysely if a DB is needed.
4. Address-first, then org/contact normalization structured on top of it.

Research grounding (3 adversarially-verified deep-research passes)

Key findings that shaped the build: (1) FS + EMexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. trains a calibrated matcher with ZERO labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. — the exact wall the original effort hit. (2) Config dominates the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — so the matcher is tunable and we didn't chase one modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. (3) Geography as the primary blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking. key is documented production practice (Grab, Geo-ER). (4) Geocode quality weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. the distance evidence (NAACCR precedent). (5) Pairwise scores are non-transitive, so clusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. is mandatory. (6) No mature TS/JS ER lib exists — greenfield. (7) Names: rule-based positional parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. + separate particle; nickname = fuzzy agreement level not a rewrite; Jaro-Winkler + edit/LCS fallback; TF-adjust = leave m, lower u for rare values, computed on-the-fly. (8) Org-name matching is a known evidence gap (PART B — cleanco/GLEIF/acronym/TF-IDF unverified) — our org canonicalizer is a solid baseline; org matching needs a follow-up pass. Full detail in project-record-matcher.md memory.

How to work here (gotchas)

• Worktree push needs an explicit refspec — the local branch is worktree-record-matcher but the remote is feat/record-matcher-foundation: git push origin worktree-record-matcher:feat/record-matcher-foundation.
• Per-package verify (the worktree is fully yarn installed and buildable): node_modules/.bin/vitest run <pkg>/, tsc -b <pkg>/tsconfig.json, prettier --write -u <pkg>/. Use the parent checkout's binaries at /home/lab/Projects/mailwoman/node_modules/.bin/.
• After adding a workspace: add it to root package.json workspaces, tsconfig.json references, vitest.config.ts aliases, then yarn install (updates the lockfile — keep it clean for CI's --immutable).
• The heavy geocoder (weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. + situssitusThe physical site address of a property, as opposed to the owner's mailing address. Parcel records often carry both; the divergence is a real-world data-quality challenge./interp shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.) is NOT in this worktree, so the real end-to-end geocode run can't be unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise.-tested here — it's operator-verifiable via the CLI. That's why geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. is an injected seam.
• Another agent was active on eval/oa-offmap-pull in the shared checkout during this shift — we stayed isolated in the worktree.
• Docs under docs/articles/ are linted as MDX — backtick any raw angle brackets or braces in prose, or the pre-commit hook rejects the commit.

Open work (prioritized)

1. CLI command mailwoman registry <csv> — construct the real geocoder (neural parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. + resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. + shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.) and inject it into geocodeAddressVia, then run ingest, resolve, GeoJSON on a dataset. Lives in mailwoman/ (the CLI package, which already has geocode-core.ts). This is the operator-verifiable integration that makes it run on real clinic data — grades the thesis against truth.
2. PR hygiene — split #607 into reviewable PRs (it bundles #599/#600/#601/#604), run CI.
3. LLM column-mapping (#603) — infer the ColumnMapping from a header + sample rows.
4. #600 tails — vendor carltonnorthern/nicknames (nickname agreement level), a Contact record type, and the org-matching follow-up (acronym/expansion, DBA/alias, TF-IDF n-gramn-gramA contiguous run of n tokens (a bigram is 2, a trigram is 3). Counting n-grams is a simple, pre-neural way to model which token sequences are common. — needs a PART-B research pass).
5. formatter follow-ups — suffix/directional expansion via @mailwoman/codex, own templates (drop @fragaria).
6. ClusteringclusteringThe final stage of entity resolution: resolve non-transitive pairwise match decisions (A↔B, B↔C, but not A↔C) into canonical entities via union-find with path compression. Each cluster of records becomes one resolved entity. refinement — centroid-linkage to damp connected-components over-merge (currently mitigated by geo-local blockingblockingThe first stage of entity resolution: generate candidate record pairs using cheap, high-recall keys (geo cell, canonical address, phone) instead of comparing every record to every other (O(n²)). The matcher only scores pairs that survive blocking.).

The 14 commits

6576e4a2 docs(concepts): the north-star strategy
3047ee7f feat(formatter): scaffold @mailwoman/formatter (#599)
b8c8a803 refactor(corpus): consume @mailwoman/formatter (#599)
d9182b12 feat(record): PostalAddress address-baseline (#600)
12701c27 feat(record): person-name parser + org-name canonicalizer (#600)
082e7db1 feat(match): Jaro/Jaro-Winkler comparators (#601)
9f9c1bbf feat(match): Fellegi-Sunter scorer (#601)
da587864 feat(match): EM estimation — label-free (#601)
7360a728 feat(match): term-frequency adjustment (#601)
330615b7 feat(match): geo-first blocking (#601)
fedc5917 feat(match): clustering (#601)
8b4e0213 feat(match): distance comparison (#601)
6659f61b feat(registry): resolveEntities + GeoJSON (#604)
d29ebec2 feat(registry): ingest layer (#604)

Plus the strategy doc docs/articles/concepts/geocode-first-record-matching.mdx (draft: true). Memory: project-record-matcher.md.