corpus-v0.4.0 Generation

How corpus-v0.4.0 is produced from corpus-v0.3.0. This is Thread B of the PHASE_8_v0_5_0_fresh_slate plan: synthetic adversarial data generated by DeepSeek and folded into the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. as a pure adapter addition. Pairs with concepts/the-knowledge-ladder for the conceptual framing and with OPERATIONS.md for the working norms.

Scope

corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-v0.4.0 = corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-v0.3.0 + DeepSeek-generated rows. The v0.3.0 shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. are not re-emitted; the v0.4.0 MANIFEST points at the same on-disk parquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. files plus the new kryptonite shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.(s).

Two row classes are in scope for the v0.5.0 plan. Both are now shipped — kryptonite in Thread B's commit (d8a6bae), transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. in this commit (Thread B2).

Class	Status (corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-v0.4.0)	Target row count	Adapter source id
Kryptonite	Shipped (Thread B, commit d8a6bae)	~5,000	deepseek-kryptonite
TransliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.	Shipped (Thread B2)	~50,000–75,000	deepseek-translit-<scrp>

The kryptonite slice is what unblocks StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 5 reconcile (Thread D) and the StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2.5 kind-classifier's joint-decoding test surface. TransliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. is what unblocks Thread A's <5% byte-fallback target on non-Latin scripts. See PHASE_8 §B for the threading rationale.

PHASE_8 §B originally enumerated eight scripts (CJK + Cyrillic + Armenian + Greek + Arabic + Hebrew + Devanagari + Thai). Thread B's smoke testing validated the prompt + alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. against five (cyrl, jpan, hans, hang, armn); those five ship in Thread B2. The remaining scripts (grek, arab, hebr, deva, thai) are additive — extending TRANSLIT_SCRIPTS + KNOWN_SOURCE_PREFIXES is sufficient — but the prompt + substring invariant should be re-smoked per-script before a production run, so they are deferred.

Pipeline

corpus-python/scripts/generate_deepseek_corpus.py
  │  prompts DeepSeek → raw JSONL (one chat-completion per batch)
  │  validates substring-match invariant on each row
  │  persists raw HTTP payload alongside canonical row (reproducibility)
  ▼
corpus-v0.4.0/kryptonite/canonical-kryptonite.jsonl     (canonical rows)
corpus-v0.4.0/kryptonite/raw-deepseek-kryptonite.jsonl  (raw API responses)
corpus-v0.4.0/kryptonite/.kryptonite-checkpoint.json    (resumable batches)
  │
  │  corpus/scripts/build-kryptonite-shard.ts
  │    streams JSONL → alignRow (BIO labels) → parquet shard
  │    composes MANIFEST = v0.3.0.shards + new shard
  ▼
$MAILWOMAN_DATA_ROOT/corpus/versioned/v0.4.0/corpus-v0.4.0/
  ├── MANIFEST.json
  └── train/part-0000.parquet          ← new kryptonite shard

Generator (corpus-python/scripts/generate_deepseek_corpus.py)

Model + API contract

Knob	Value
Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'	deepseek-v4-flash
Reasoning effort	low
Endpoint	https://api.deepseek.com/v1/chat/completions
Max tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.	20,000 per response
Retries	5, exponential backoff at 2.0× from 2s
Timeout	300s per call

The OpenAI-compatible Chat Completions schema is used directly via stdlib urllib — no SDK dependency. 429 / 5xx are retried; 4xx (other) bubble out as fatal.

Concurrency + rate

Default --concurrency 15, --batch-size 50. Wall-clock observation on the production run: ~12 rows/s sustained, dominated by DeepSeek server latency rather than client throughput. 5,000 rows takes ~7 minutes. No rate-limit headers are surfaced; if the API ever returns 429 the backoff loop handles it.

License hygiene

Raw DeepSeek API responses are AGPL-compatible for the operator's use case (no output-source-attribution constraints from DeepSeek's terms as of 2026-05-23). Every emitted row carries:

• license: "Synthetic (DeepSeek-v4-flash, AGPL-compatible)"
• synth.method: "deepseek-kryptonite:<category>" (or deepseek-translit:<script>)
• synth.base_source_id: "kryptonite-seed:<category>" (or the seed row's source_id)

source: "deepseek-kryptonite" (or "deepseek-translit-<slug>") makes provenance explicit at audit + downstream-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time so the model cardmodel cardA JSON metadata file (model-card.json) shipped with each weights bundle. It declares the model version, lineage, label set, required inference channels (anchor, gazetteer), calibration data, and training provenance. can disclose synthetic fraction.

Reproducibility

Three artifacts pin the generation:

1. Prompts — the system + user prompt strings live in generate_deepseek_corpus.py constants (KRYPTONITE_SYSTEM, KRYPTONITE_USER_TEMPLATE, TRANSLIT_SYSTEM, KRYPTONITE_CATEGORIES). Version-pinned through git history.
2. Raw responses — every chat-completion is persisted to raw-deepseek-kryptonite.jsonl with {batch_id, category, n_requested, model, finish_reason, usage, response_content}. Regenerating from this raw log requires only the JSON validator, not another DeepSeek call.
3. CheckpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. — .kryptonite-checkpoint.json records the set of completed batch_ids. Reruns with identical args skip done batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU..

batch_id is a deterministic SHA-256 prefix over (category, batch_index, n), so re-running the generator with the same args produces the same batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. ids — the checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. set composes with itself across runs.

Kryptonite categories

The generator's KRYPTONITE_CATEGORIES list defines 10 adversarial flavours, each with a category id, a description, 2-3 hand-written seed examples, and a weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. that allocates the total budget proportionally:

Category	Description	WeightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.
venue-shadow-region	VenuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. brand contains a regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.-like tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.; actual regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. is elsewhere	1.0
locality-shadow-country	US localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. shadows a famous foreign citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. (Paris TX, Moscow ID)	1.0
mid-position-postcode	PostcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. appears between localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. and countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., not at the end	1.0
repeated-token	Same tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. in venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. and localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. (Buffalo Buffalo, Walla Walla)	0.9
abbrev-collision	StateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. abbreviation collides with a venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label./streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.	0.8
saint-shadow	Saint X / St. X colliding with a famous European saint-name citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.	0.8
compass-prefix	Compass-prefixed localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. whose base name is a famous other place	0.7
region-shadow-venue	VenuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. brand embeds a US stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. name as a tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.	0.7
french-saint	FR equivalent of saint-shadow	0.7
po-box	PO BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. intermixed with streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-style tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.	0.5

The substring-match invariant (every component value must appear verbatim in raw) is enforced in the system prompt and revalidated locally before the row is committed to the canonical JSONL. Failure rate observed on production run: ~0.02%.

Builder (corpus/scripts/build-kryptonite-shard.ts)

Streams the canonical JSONL through alignRow (corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus./src/align.ts) to produce tokens + labels, writes a single parquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. under train/, and emits the combined MANIFEST.json. v0.3.0 shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. descriptors are copied verbatim into the new manifest — no v0.3.0 bytes are touched. New shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. descriptors are stamped with source: "deepseek-kryptonite" so corpus-audit can attribute them without the filename-prefix inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. fallback.

Quarantined rows (alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. failures) are logged to quarantine-kryptonite.tsv alongside the new shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.. Surface-form validation already happens in the Python generator, so the alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. step should reject <1% — anything above 5% indicates a prompt or substring-match regression and should be investigated before commit.

Split policy

All kryptonite rows land in train. Synthetic adversarial data is augmentation; it must not appear in val or test where it would inflate the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. against itself. The v0.3.0 splitter's localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-holdout policy (corpus/src/split.ts) does not apply — kryptonite rows have no natural localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. boundary and are not produced by the holdout regionsregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. anyway.

The kryptonite catalogue's evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. surface lives elsewhere — StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 5 reconcile (Thread D) ships its own hand-curated fixture set. See PHASE_8 §D.

Auditing

npx tsx corpus/scripts/audit.ts \
  $MAILWOMAN_DATA_ROOT/corpus/versioned/v0.4.0/corpus-v0.4.0 \
  --config corpus-python/src/mailwoman_train/configs/v0_5_0.yaml

The audit reports per-source shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. counts and (with --config) effective sample weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.. deepseek-kryptonite is now in KNOWN_SOURCE_PREFIXES. Expected: 1 train shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. with source: "deepseek-kryptonite", sub-1% of total shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. (the v0.3.0 baseline has 674 train shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.; one more is noise at the audit level).

Transliteration generation (Thread B2)

Goals

• Seed corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.: /data/corpus/versioned/v0.4.0/staging/seeds-en-us.jsonl (~4.5K US rows) and seeds-fr-fr.jsonl (~10.5K FR rows), already sampled from the v0.3.0 train set by the previous session.
• For each seed, generate one transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. per target script. Five scripts:
  • cyrl — Russian Cyrillic (localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. ru-RU)
  • jpan — Japanese Katakana + Kanji (localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. ja-JP)
  • hans — Simplified Chinese (localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. zh-CN)
  • hang — Korean Hangul (localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. ko-KR)
  • armn — Armenian (localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. hy-AM)
• Target row count: 15K seeds × 5 scripts = ~75K transliterationstransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.. The original plan said ~50K; the seed pool grew during sampling and the upper bound is what fits the v0.5.0 byte-fallback budget on Thread A.

Mode + invocation

Implemented in generate_deepseek_corpus.py:

python3 corpus-python/scripts/generate_deepseek_corpus.py \
  --mode transliteration \
  --out-dir /data/corpus/versioned/v0.4.0/transliteration \
  --seed-paths /data/corpus/versioned/v0.4.0/staging/seeds-en-us.jsonl \
               /data/corpus/versioned/v0.4.0/staging/seeds-fr-fr.jsonl \
  --scripts cyrl jpan hans hang armn \
  --batch-size 50 --concurrency 15

The Thread B2 production run consumed the full seed pool (14,978 rows = 4,478 en-US + 10,500 fr-FR). 5 scripts × 14,978 seeds = 74,890 planned rows. Wall-clock and rejection rate are pinned in the Changelog below.

Prompt design

TRANSLIT_SYSTEM in the generator script — the contract is:

• Keep digits / commas / periods / hyphens verbatim.
• Transliterate place namestoponymA proper name for a geographic place. + streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-type words using natural conventions for the target script; do not translate semantically.
• Mirror the input component tagscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. exactly (every input tag appears in output).
• Surface-form invariant holds: every component value substring-matches the transliterated raw.

Each batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. is built per-script (not per-seed): one chat completion handles 50 seeds in one target script. The response is JSONL keyed by i (batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. index 0..49). Substring validation rejects malformed transliterationstransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. before commit; rejection rate in initial prompt-engineering smoke was ~3% — acceptable, but the prompt is worth tuning before the production 75K pass.

Cost + wall time estimate

• 75K rows / 50 per batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. = 1,500 batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU..
• Per-batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. latency is API-dominated; with reasoning headroom (see below), 30–60 s.
• Wall time at conc=15: ~1,500 batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. × 45 s / 15 ≈ 75 min on a clean run.

max_tokens and the reasoning budget

DeepSeek-v4-flash with reasoning_effort=low still consumes a non-trivial reasoning budget on transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.. Thread B2's first launch ran at max_tokens=20000 (the same knob that worked for kryptonite) and saw 30/32 batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. hit finish_reason=length because reasoning ate 12–15K of the 20K budget, leaving <5K for the 50-row JSONL output. Truncated responses produced ~8 rows per batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. instead of 50.

The validated production knob is max_tokens=60000: empirically reasoning peaks around ~15K and 50-row output uses ~5K, so 60K leaves comfortable headroom. The Thread B2 generator also marks finish_reason=length batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. with a !RETRY: prefix so they don't checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. and get retried on subsequent runs — defence in depth for the rare batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. that still truncate at 60K.

This budget reasoning is transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.-specific; the kryptonite mode at max_tokens=20000 remains correct because English-only ASCII output uses ~1 tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. per row character and 50 rows fit comfortably.

What Thread B2 adds

1. Runs the transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. generator end-to-end against the full seed pool.
2. New build-transliteration-shard.ts (now corpus/src/tools/shard-translit.ts) mirroring build-kryptonite-shard.ts — buckets canonical rows by source and emits one parquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. per script (train/part-translit-<slug>.parquet).
3. Composes the new MANIFEST as (v0.3.0 base shards) + (Thread B kryptonite shard) + (Thread B2 translit-<slug> shards).
4. Migrates v0.3.0 shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. descriptors in MANIFEST from $MAILWOMAN_DATA_ROOT/... to /data/..., the form that container loaders resolve at trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time. The on-disk bytes are unchanged; only the path strings in MANIFEST move. (Thread B's MANIFEST mixed the two forms; this PR canonicalizes.)
5. KNOWN_SOURCE_PREFIXES in audit.ts already contains the five deepseek-translit-* slugs from Thread B — no audit.ts edit needed.

Known risks

• Cross-script seed pollution. If the seed pool is biased (e.g. all FR seeds are Île-de-France because the BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. sampling skewed there), the resulting transliterationstransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. would teach the tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. about that bias 5× over. The current staging files were sampled stratified by source only. RegionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. stratification is deferred — accepted as a known limitation of this pass; can be revisited if Thread A's byte-fallback evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. surfaces a regional skew.
• Whitespace-tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. interaction with CJK. AlignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. uses the whitespace tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses., which treats space-less CJK as a single tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.. The substring invariant still holds (raw and component values are space-matched verbatim), so alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. passes — but per-character labelling at trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. time comes from the sentencepieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. Thread A retrains, not from the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens..

Changelog

Date	Change
2026-05-23	Initial doc + kryptonite slice generation (5K rows, deepseek-v4-flash).
2026-05-24	TransliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. slice generation shipped under Thread B2: 73,319 rows from 5 scripts × 14,978 seeds (en-US + fr-FR), max_tokens=60000, 5.2 rps sustained, ~234 min wall-clock at conc=15. 73,316 rows aligned (3 quarantined). 5 shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. added: part-translit-{armn,cyrl,hang,hans,jpan}.parquet. v0.3.0 shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. paths in MANIFEST canonicalized from $MAILWOMAN_DATA_ROOT/... to /data/....