Corpus construction

The trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. is the largest single source of leverage in the project. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' can only learn patterns that appear in the data. This article walks through how Mailwoman builds its corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. — what sources are in it, how rows are aligned to BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components., and how the synthesis step multiplies the effective size.

A general overview of the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. is in Training pipeline. This article zooms into the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. side (StagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 1–4 of that pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.).

What "the corpus" is

A corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. version is a frozen set of (raw, tokens, BIO labels) triples ready for trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.. The current version is corpus-v0.3.0:

metric	value
total aligned rows	677 million
quarantined rows	214,118 (0.03%)
active source adapters	11
disk size (output)	~30 GB ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet.
disk size (with intermediates)	~430 GB
wall-clock to build	~6.5 hours
splits	train (90%) / val (5%) / test (5%)

The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. is built once per ship. Re-running the build is idempotent; running incremental updates is not yet supported.

The data sources

The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. draws from many open data sources. Each one contributes a different kind of signal:

adapter	rows in v0.3.0	what it contributes
WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. admin	4.2M	countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. / regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. / localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. strings + their canonical names
WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. postalcode	104K	postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. → localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. mappings (US-heavy)
BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. (Base Adresse Nationale)	27.1M	every French address, house-number level
TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data. (US Census)	23.3M	US streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. segmentssegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context. + place namestoponymA proper name for a geographic place.
NPPES	9.2M	US healthcare provider names + business addresses (venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. + streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. level)
HRSA-FQHC	19K	US federally-qualified health centres
IMLS-PLS	18K	US public libraries (venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. + address)
NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. (US DOT)	57.9M (new in v0.3.0)	structured 911-grade address pointssitus dataA dataset of exact address-point coordinates (rooftop-level). Mailwoman's geocoder uses a national situs layer (124.9M US points built from state address-point sources) as the highest-precision tier of the geocode cascade., full venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label./streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./house_number coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.
stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-ny-notaries	32K	NY notary names + business addresses
stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-tx-notaries	511K	TX notary names + business addresses
stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-ia-contractors	17K	Iowa contractor names + addresses

Skipped in v0.3.0:

• openaddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. — waiting on per-row licence filter (PR #56).
• fcc-bdc — license incompatibility (CostQuest gray-copyright).
• usgov-samhsa-treatment-locator — source bulk export went offline (PR #113 deferred the adapter).

The diversity of sources matters more than the absolute count. NPPES contributes 9.2M rows but it is the single largest source of venue + address co-occurrence. Without it, the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' would not have learned that a venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. precedes a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. more often than it follows one.

Stage 1 — Per-adapter row emission

Each adapter is a TypeScript async generator that yields CanonicalRow objects. The contract is straightforward:

interface CorpusAdapter {
	id: string // 'usgov-nppes'
	defaultLicense: string // 'Public Domain'
	description: string
	rows(opts: AdapterOptions): AsyncIterable<CanonicalRow>
}

The adapter does the localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for.-specific work: column name mapping, address composition (when the source has street1 + street2 instead of a single street), source-specific cleaning (NPPES drops the Provider Other Last Name column on output, BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. composes numero + rep for house_number, etc.).

Output is streamed to intermediate/<adapter>/raw.jsonl.gz. Streaming matters — some adapters emit tens of millions of rows.

Stage 2 — Alignment

The alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. step takes a (raw, components) pair and produces a (raw, tokens, BIO labels) row. Two cases:

Success case. Every component value is found inside raw. The aligner tokenizes raw, marks the first tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. of each spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. as B-tag, the rest as I-tag, and the rest as O.

Failure case. A component value cannot be located in raw. The row goes to quarantine with a reason: component-not-found:locality, edit-distance-exceeded:postcode:7, etc. Quarantined rows are written to quarantine/<reason>.jsonl.gz for later inspection but do not go to trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input..

The aligner uses fastest-levenshtein for fuzzy matching with a tunable threshold (default 2 edits). This catches "Pennsylvania" vs "Pennsylvana" but rejects more aggressive mismatches.

Stage 3 — Synthesis (the multiplier)

Real-world addresses appear in many surface forms — abbreviated, capitalised, punctuated, separated. To make the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' robust to these variations, Mailwoman runs a synthesis pass over the aligned rows.

For each aligned row, synthesis generates several surface-form variants by applying combinations of:

• StateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation (California ↔ CA)
• Comma drop (New York, NY 10118 → New York NY 10118)
• Casing changes (upper, lower, mixed)
• Accent stripping (Île → Ile)
• Whitespace normalisation (collapse multiple spaces, add tab separators)
• Street suffixstreet affixA modifier on a street name indicating type or direction — Street, Avenue, rue, Calle, N, East. Mailwoman tags these as street_prefix / street_suffix, recognized via a morphology FST. abbreviation (Avenue ↔ Ave)

The BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components. are recomputed for each variant (because tokenisation may differ — "NY" is one tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., "New York" is two). Variants whose alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. fails get quarantined.

Synthesis is the reason v0.3.0's 677M aligned rows came from only ~120M source rows. The multiplier is about 5x.

Synthesis is on by default but can be disabled (synthesize: false in the build options) for adapter debugging where you want clean per-adapter output.

Stage 4 — Splits

Once aligned + synthesized, rows go through the split step. Three groups: train (~90%), val (a few percent for in-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.), test (held out).

The split is localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-aware: rows that share a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. stay in the same split. This is what keeps evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits.-free. Without it, a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' could memorize "Brooklyn = localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy." from a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. row and recover it on a val row, looking better than it is.

The split assignment is written to SPLIT_MANIFEST.json — every later step can reproduce which rows went where. If the same input goes through two different build runs, it ends up in the same split (deterministic hash on source_id).

Stage 5 — Sharding

Final output is ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. files with about 1 million rows per shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.. ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. is columnar — the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data loader streams one column group at a time without loading the whole file. Total v0.3.0 output: ~30 train shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. + 1 val + 1 test = 32 files, ~30 GB.

The trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. picks up from here. See Training pipeline for what happens next.

What corpus building is good at

• Reproducibility. Same sources + same code = same corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., byte for byte. Every aligned row's source_id makes it traceable back to the upstream row.
• Quarantine for diagnostics. The quarantine pile makes data-quality issues visible. A spike in quarantine rate for a specific adapter is the first sign that the source's schema changed.
• Per-source weightingsource weightingMultiplicative per-source weights applied during training to oversample underrepresented corpus sources, balancing a corpus dominated by a few large datasets. at trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time. The trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data loader supports source_weights to oversample under-represented sources. v0.2.0 used this to fix a positional-heuristic overfit where the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' had learned "first tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. = localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy." because 75% of v0.1.0 corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. came from wof-admin.

What corpus building is bad at

• Incremental updates. Today, adding a new source means a full rebuild. ~6 hours of wall-clock.
• Wall-clock cost scales with synthesis. The 5x synthesis multiplier doubles the alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. + sharding time vs. a synth-off build. v0.3.0 added more sources without bumping max_steps, so the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. is now under-utilized: a 50K-step trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run sees only 6.4 million examples (~1% of the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. per pass).
• Disk pressure. The intermediate NDJSON dumps land at ~400 GB and stay until the next build. The host is on a 1 TB nvme; this is fine but tight.

Where this lives in the code

• Adapters: corpus/src/adapters/<source>/adapter.ts
• Aligner: corpus/src/align.ts
• Synthesis: corpus/src/synthesize.ts
• Splitter: corpus/src/split.ts
• Sharder: corpus/src/parquet.ts
• Build orchestrator: corpus/src/build.ts + corpus/scripts/run-corpus-build.ts
• Schema reference: corpus/src/types.ts (CanonicalRow, LabeledRow, QuarantinedRow)
• Plan doc: PHASE_1_corpus.md

See also

• Training pipeline — what happens after corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. build
• BIO labels — what the alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. step outputs
• Tokenization — how raw becomes tokens