Corpus v0.5.0 rebuild plan (2026-06-11 — scope frozen by the #519 review)

The four passengers, ruled and locked: char-offset labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. (#519, from-source, parallel arrays), DE holdout (already merged into defaultHoldouts() — activates automatically), ZCTAZCTA (ZIP Code Tabulation Area). The US Census Bureau's polygon approximation of a ZIP code, generalized from census blocks rather than USPS delivery routes. Explicitly not USPS ground truth and often wrong in rural areas. centroid fill (#525), FTS alias-bag boundary separator (#523). Everything else is v0.5.1. This doc is the integration order + the estimate scaffold; it becomes the build runbook when scheduled.

Order of operations

1. Format change, code-first (corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus./ + corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-python, no build yet):

   • alignRow emits span_starts[]/span_ends[]/span_tags[] (deleting the tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. quantization — the char offsets already exist internally). Build-time NFC assertion per row (the ruling replaced the converter's Unicode class with this).
   • encode_row consumes spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. directly (promoting the internal per-char array to the input contract); the glue augmentation and #511 relabel pass re-target spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. (both simplify — the relabel becomes char arithmetic).
   • Per-piece channel invariance gates: anchor + gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. painting asserted identical on a fixture set of punctuation-free rows (the consult keeper, now an encode-time test).
   • Every shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. builder migrates (affix, unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise., intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model., po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise./cedexCEDEX (Courrier d'Entreprise à Distribution Exceptionnelle). A French postal routing for high-volume business mail: a CEDEX code delivers directly from a sorting centre, bypassing the local post office. A common negative-space format Mailwoman must parse.+au/nz, countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., german). Builder audits move to RAW-surface comparison (the dotted-blind-spot lesson).

2. ZCTAZCTA (ZIP Code Tabulation Area). The US Census Bureau's polygon approximation of a ZIP code, generalized from census blocks rather than USPS delivery routes. Explicitly not USPS ground truth and often wrong in rural areas. join (#525): Census ZCTAZCTA (ZIP Code Tabulation Area). The US Census Bureau's polygon approximation of a ZIP code, generalized from census blocks rather than USPS delivery routes. Explicitly not USPS ground truth and often wrong in rural areas. gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. file → build-pilot-anchor-lookup.py + the WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. row builders, source provenance per filled row. Acceptance: anchor placeholder rate 38% → single digits; "90210" circles in the demo.

3. FTS separator (#523): boundary-preserving separator in buildPlaceSearchFts's GROUP_CONCAT + the matching exact-tiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. check; slim-DB rebuild rides the same artifact pass.

4. Pilot build (1 shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., ~1M rows): calibrates the full-build wall-clock estimate AND runs the invariance gates against real data before committing days of compute. THE GO/NO-GO GATE.

   RAN 2026-06-12 — verdict: GO, after two aligner fixes the gate itself caught. Three iterations (TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data., 150k canonical → ~312k labeled, 36 s each):

   • pilot1: 270 rows (0.088%) with corrupted spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. — locateSpan's leftmost-substring match let regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. "AK" claim the inside of "Umak"/"Lake". Fix: boundary-aligned matches preferred, intra-word kept as fallback (the German-affix sub-word case is essential).
   • pilot2: 0 corrupted; 21 quarantines remained — the homonym-steal class ("Alaska Regional Dr, Alaska": regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. claimed the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.'s word). Fix: longest-value-first location.
   • pilot3: 311,750 rows, 0 quarantined, 0 violations; the python loader streamed the shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. with spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. valid end-to-end.

   Wall-clock calibration: 36 s per 150k canonical single-process → full ~677M labeled ≈ 22 h one process; embarrassingly parallel by adapter, so a same-day build at modest parallelism.

5. Full from-source build (~677M rows + the v0.4.x overlay shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. re-emitted natively). DE holdout materializes here (SPLIT_MANIFEST picks up Saarland + Mecklenburg-Vorpommern).

6. Validation campaign: invariance on sampled converted-equivalent rows, lint-corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., a DE honest-evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. slice trust check (≥1000 rows), then the first new-format trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run with the PRE-REGISTERED bridge-retirement gate (dotted po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. ≥ 89.1 bridge-OFF, FR postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. ≥ 99.5, affix floors unchanged, over-merge precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. floor, #518 as the punctuation lens).

Estimates (scaffold — the pilot build firms these)

Step	Type	Estimate
Format change + builders + gates	agent-nights	2–3
ZCTAZCTA (ZIP Code Tabulation Area). The US Census Bureau's polygon approximation of a ZIP code, generalized from census blocks rather than USPS delivery routes. Explicitly not USPS ground truth and often wrong in rural areas. join + FTS separator	agent-night	1
Pilot build + calibration	wall-clock	hours (measured, then extrapolated)
Full build	wall-clock	days-scale; the pilot decides; embarrassingly parallel if needed
First new-format train + gate	A100 + battery	~$8–10, one evening (the v4.4.0 pattern)

What this unblocks, in order

Punctuation supervision (the bridge's retirement, #518's weak quadrants), the paired-delimiter spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. proposer (#518's revival verdict — its proposals become directly storable supervision), and the conventions lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.-mask's full-run ride (banked since night-11, FR-regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. floor ≥ 25.6).