DE-inclusive corpus assembly (2026-06-04)

The data step that gates the PR3 self-conditioningself-conditioningAn auxiliary model head that predicts the input's locale (language/country) and then FiLM-modulates the encoder so parsing adapts to the detected locale. Structural facts like word order still need explicit training on both orders. pilot (plan). The Pilot A config wanted a US/FR/DE corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. from scratch; v0.4.0 is US/FR only. This records how the German rows were added and how to deploy them.

What was built

A new overlay corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus., v0.4.1-de: v0.4.0's 684 base shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. (US/FR, referenced verbatim by their ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. /data paths) plus two new German shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row..

shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.	split	rows	source
part-german-train.parquet	train	200,000	synth-german
part-german-val.parquet	val	4,000	synth-german

The German rows are real OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. Berlin + Saxony tuples (~1.2M unique, cached on disk) rendered in German order (house-number after streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. before citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.) by synthesize-german.ts, then aligned to BIO. The build reuses the v0.8.0 scripts/build-german-shard.mjs precedent — but at 200K rows, 40× the v0.8.0 supplement, because that run was a continue-train supplement (0.2 weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. on an already-trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.') while this is a from-scratch pilot that has to learn German from zero. Align pass rate was ~99.9% (186 skipped of 200K).

Why this is the right data for the pilot, not a repeat of v0.8.0

The v0.8.0 German order-shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. failed — German streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels./house-number F1 jumped (+22/+16pp), but localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. and postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. F1 collapsed (−37/−58pp) via end-of-string spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. fragmentation, and US/FR slipped past the 1pp tripwire. That collapse is exactly the failure self-conditioningself-conditioningAn auxiliary model head that predicts the input's locale (language/country) and then FiLM-modulates the encoder so parsing adapts to the detected locale. Structural facts like word order still need explicit training on both orders. is designed to prevent: a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' that has resolved "this is a German address" globally, before per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., shouldn't bleed a citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.'s lead tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. into the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.. So this corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. + the PR3 architecture is the test of that hypothesis, and the cross_pollution tripwire (now with the 4K German val shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. for support) is the live readout of whether it holds.

The mix knob

In v0.9.0-pilot-selfcond.yaml, synth-german is weighted 6.0 of a ~34 source-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. total — German at ~18% of the mix. Enough for a from-scratch modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' to learn it, inside the synthesis-as-supplement guideline. It is the key tunable: raise it if DE localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. F1 lags the 70% gate, lower it if US/FR slip past the 1pp tripwire.

Reproduce + deploy

Built locally at $MAILWOMAN_DATA_ROOT/corpus/versioned/v0.4.1-de/corpus-v0.4.1-de/. Validated: the data loader reads country=DE / locale_id=2 rows from both splits. To deploy to the ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume the trainer reads:

node scripts/build-german-shard.mjs --output /tmp/german-train.jsonl --count 200000 --seed 42
node scripts/build-german-shard.mjs --output /tmp/german-val.jsonl   --count 4000   --seed 99
python3 scripts/jsonl-to-parquet.py --input /tmp/german-train.jsonl --output <NEW>/train/part-german-train.parquet
python3 scripts/jsonl-to-parquet.py --input /tmp/german-val.jsonl   --output <NEW>/val/part-german-val.parquet
python3 scripts/assemble-de-overlay-manifest.py --base <v0.4.0 MANIFEST> --new-dir <NEW> \
  --modal-root /data/corpus/versioned/v0.4.1-de/corpus-v0.4.1-de
modal volume put mailwoman-training <NEW>/train/part-german-train.parquet corpus/versioned/v0.4.1-de/corpus-v0.4.1-de/train/part-german-train.parquet
modal volume put mailwoman-training <NEW>/val/part-german-val.parquet     corpus/versioned/v0.4.1-de/corpus-v0.4.1-de/val/part-german-val.parquet
modal volume put mailwoman-training <NEW>/MANIFEST.json                    corpus/versioned/v0.4.1-de/corpus-v0.4.1-de/MANIFEST.json

After the upload, Pilot A is a one-command launch (modal run -d scripts/modal/train_remote.py --config v0.9.0-pilot-selfcond.yaml --resume none --trackio), against a tripwire already wired to report at 20k whether self-conditioningself-conditioningAn auxiliary model head that predicts the input's locale (language/country) and then FiLM-modulates the encoder so parsing adapts to the detected locale. Structural facts like word order still need explicit training on both orders. earns the full run.