Skip to main content

Phase 1 — Corpus Pipeline

Goal: build a reproducible, versioned dataset pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.. End stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. is corpus-v0.1.0 on disk, containing aligned BIO-labeled ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. for US + FR, with a hand-labeled golden evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set.

Duration estimate: 2 weeks.

Branch: neural/phase-1-corpus

Depends on: PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 0 complete and tagged.

Pre-flight​

  • PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 0 success criteria all green
  • Disk space available at /data/corpus/ per reference/OPERATIONS.md layout
  • You understand BIO labeling and the LabeledRow shape

Tasks​

1. Package scaffolding​

  • Create packages/corpus/ workspace
  • Create packages/corpus-python/ (Python, not in workspaces — has its own pyproject.toml)
  • packages/corpus/src/types.ts — CanonicalRow, LabeledRow, CorpusAdapter, AdapterOptions per reference/INTERFACES.md
  • Add to root tsconfig references, root package.json scripts

2. Adapter framework​

  • packages/corpus/src/adapter.ts — base CorpusAdapter interface, helper utilities (streaming, checksum, dedup)
  • packages/corpus/src/runner.ts — drives an adapter, writes intermediate JSONL, reports progress, handles backpressure
  • CLI: npx mailwoman corpus run <adapter-id> --input <path> --output <dir> [--country XX] [--limit N]

3. Adapters — priority order​

Build adapters in this order. Each must have:

  • A small fixture in packages/corpus/fixtures/<adapter-id>/ (license-clean, hand-crafted, < 1MB)
  • An integration test that runs the adapter against the fixture
  • A README documenting the source, license, download instructions, expected schema, known quirks

3a. wof-admin (P1, coarse)​

  • Input: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. SQLite distributions (whosonfirst-data-admin-us-latest.spatial.db, -fr)
  • Emit one row per WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. record, with synthesized raw strings using countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-appropriate templates
  • Walk the ancestry chain — emit hierarchical variants (citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. alone, citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.+regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.+regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.+countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.)
  • License: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. is CC0
  • Expected output: ~50k US localities, ~36k FR communes, plus regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality./countries

3b. wof-postalcode (P1, coarse)​

  • Input: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. postalcode SQLite distributions
  • Emit rows pairing postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. with its parent localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.
  • License: CC0

3c. osm-places (P1, coarse corroboration)​

  • Input: OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. PBF file, filtered to place=city|town|village|hamlet|suburb|neighbourhood
  • Emit rows for each named placevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label., with admin hierarchy from is_in:* tags or reverse-geocoded from WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.
  • License: ODbL — tag in license field
  • Use osm-pbf-parser-node or equivalent (verify maintenance before committing)

3d. address-formatting (P1, synthesis support)​

  • Vendor or fetch OpenCageData's address-formatting repo (MIT). It contains countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-keyed templates for rendering component dicts into strings.
  • packages/corpus/src/format.ts exposes formatAddress(components: ComponentDict, country: string): string
  • This is not an adapter — it's a utility used by adapters and synthesis to render raw from component dicts.

3e. ban (P2, FR street-level)​

  • Input: Base Adresse Nationale dump from adresse.data.gouv.fr. Format: CSV, very large (~25M rows).
  • Emit rows with house_number, street, postcode, locality, region, country='FR'
  • License: ODbL / Licence Ouverte
  • Note: BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. has authoritative French addresses. This is your highest-quality FR source.

3f. openaddresses (P2, US/global street-level)​

  • Input: OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. GeoJSON, countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.-partitioned downloads
  • License: varies per source — propagate per-row
  • Emit streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-level rows

3g. osm-addr (P2, US/FR street-level)​

  • Same OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. PBF, filtered to addr:* tagged ways/nodes
  • Lower quality than OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. or BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. but broader coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.

3h. US gov registries (P3, US venue-level)​

Defer to end of PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1 or push to PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 2 if running long. Each is a small adapter, none individually critical:

  • usgov/hrsa — Health Resources & Services Administration facility list
  • usgov/npi — National Provider Identifier (medical providers)
  • usgov/fcc — FCC licensee addresses

3i. SIRENE (P3, FR venue-level)​

  • Input: SIRENE bulk download (FR business registry)
  • License: Licence Ouverte
  • Defer if running long

4. Alignment​

Given a CanonicalRow with raw and components, produce a LabeledRow with tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-level BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components..

  • packages/corpus/src/align.ts
  • Strategy: for each component value, find its character spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. in raw using fuzzy match (fastest-levenshtein). Tokenize raw with SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text.. Assign BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components. to tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. whose spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. overlap component spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree..
  • Reject rows where any component cannot be aligned (component text doesn't appear in raw within edit distanceedit distanceThe minimum number of single-character insertions, deletions, or substitutions needed to turn one string into another (Levenshtein distance). Used in corpus alignment and record matching. threshold). Write rejected rows to /data/corpus/quarantine/ with a reason for human review.
  • UnitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. tests: alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. correct on hand-crafted examples covering: missing components, reordered components, abbreviated forms, accented vs unaccented text.

⚠ Use the same SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. and inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. will use. Train it first (next task), then run alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens..

5. SentencePiece tokenizer training​

The tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. is trained on the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., not picked off the shelf.

  • packages/corpus-python/scripts/train_tokenizer.py
  • Input: a sample (say 5M lines) of raw strings from coarse adapters, balanced US/FR
  • Train SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. with: vocab_size=16000, character_coverage=0.9995, model_type=unigram, byte_fallback=true
  • Output: tokenizer.model and tokenizer.vocab written to /data/models/tokenizer/v0.1.0/
  • Run alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. using this tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses..

⚠ TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. version is locked into corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. version. corpus-v0.1.0 ships with tokenizer-v0.1.0. Don't retrain mid-corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus..

6. Synthesis / augmentation​

  • packages/corpus/src/synthesize.ts
  • Augmentations for both US and FR:
    • Case perturbation (random upper/lower)
    • Punctuation drop/add (commas)
    • Abbreviation swap (Street ↔ St using Mailwoman's existing dictionaries — reuse from resources/)
    • Whitespace normalization variants (single/double space, tab/newline)
    • Typo injection (single-char edits, low rate ~2%)
  • FR-specific augmentations:
    • Accent stripping (Hôtel ↔ Hotel)
    • Particle variants (Rue de la République ↔ Rue République)
    • Arrondissement notation (Paris 8e ↔ Paris VIII ↔ 75008 Paris)
    • CEDEXCEDEX (Courrier d'Entreprise à Distribution Exceptionnelle). A French postal routing for high-volume business mail: a CEDEX code delivers directly from a sorting centre, bypassing the local post office. A common negative-space format Mailwoman must parse. variants (with and without)
  • US-specific augmentations:
    • StateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation vs full (OR ↔ Oregon)
    • Directional abbreviation (SE ↔ Southeast)
    • ZIP+4 with and without dash
  • Each augmented row carries synth.method and synth.base_source_id.

7. Parquet output​

  • packages/corpus/src/parquet.ts — write labeled rows to ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., ~1M rows per shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.
  • Schema: raw: string, tokens: list<string>, labels: list<string>, country: string, locale: string, source: string, source_id: string, corpus_version: string, license: string, synth_method: string?, synth_base_id: string?
  • Use @dsnp/parquetjs or, if that proves limiting, write JSONL and convert via a tiny Python script (PyArrow). Either is acceptable.
  • Output path: /data/corpus/versioned/corpus-v0.1.0/

8. Eval splits​

⚠ This is where corpora silently leak. Do it correctly.

  • packages/corpus/src/split.ts
  • Split strategy: hold out by localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., not random row sampling
    • US: hold out all rows from Vermont, Wyoming, North Dakota (low-density statesregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., ensures generalizationgeneralizationA trained model's performance on data unlike its training set — new regions, new input distributions. The property honest eval is designed to measure.)
    • FR: hold out all rows from Corse, Lozère, Creuse (small departments)
  • Output: split manifests as JSON files listing source_ids in each split. Manifests go in git.
  • Splits: 90% train, 5% val, 5% test
  • Document the holdout choices in DECISIONS.md

9. Golden eval set​

  • /data/eval/golden/v0.1.0/
  • 500 US addresses, 500 FR addresses, hand-labeled by a human (the maintainer). Use the existing Mailwoman parser as a starting point and hand-correct.
  • Each entry: { raw, components: { ... }, source: 'golden', notes: '...' }
  • Cover: residential, commercial, PO boxesPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise., intersectionsintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model., venuesvenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label., edge cases (single-line, multi-line, abbreviations, typos)
  • Check into git. This is the contract for "what good looks like."

▶ Drafting the golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. takes real time. Start it in week 1 of PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1 even while building adapters. Don't leave it for the end.

10. Corpus build pipeline​

  • npx mailwoman corpus build --version 0.1.0 — single command that runs all adapters, alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens., synthesis, splits, writes ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet.
  • Reproducible — same inputs → same outputs
  • Logs progress, writes a manifest with file checksums to /data/corpus/versioned/corpus-v0.1.0/MANIFEST.json

Success criteria checklist​

  • corpus-v0.1.0/ exists on disk with ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.
  • MANIFEST.json has checksums for every shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.
  • At least 5M labeled rows (coarse + streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. where available)
  • EvalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. split manifests in git
  • Golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. in git: 1000 hand-labeled entries
  • TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. v0.1.0 saved alongside corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.
  • All adapters have fixtures + integration tests
  • npm test green
  • LOG.md, DECISIONS.md up to date
  • Branch tagged neural-phase-1-complete

Common pitfalls​

  • ❌ Splitting train/test randomly. LocalitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. holdout or it doesn't count.
  • ❌ Retraining the tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. after some alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. has been done. TokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. shift, labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. become wrong.
  • ❌ Skipping the quarantine pile. The alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. failures are where future bugs hide.
  • ❌ Forgetting to record the license per source. You'll regret this when someone asks if they can use a derived modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' commercially.
  • ❌ Letting OSMOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. dominate by row count. Stratified sampling at trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time, but warn now if one source is > 60% of rows.

When to call this phase done​

When you can run npx mailwoman corpus build --version 0.1.0 from a clean stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., get a complete corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. on disk, and run a small Python script that loads a shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. and prints (tokens, labels) pairs that look obviously correct.