v0.5.0 char-offset launch — shift postmortem (2026-06-12)

Continuation of the night-12 build session. The build finished and validated; this shift's job was to get the first modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' TRAININGtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. on the v0.5.0 char-offset corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. It did — after closing a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. gap that should have been caught earlier and routing around a ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume consistency failure. TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. completed all 40k steps and the gate ran: the bridge-retirement test PASSES (po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. 90 bridge-OFF ≥ 89.1, intrinsic). 15/17 tags hold flat-or-better actual-vs-actual; the one real casualty is fr.house_number, −8.1pp vs v4.4.0's actual 97.7 (the gate floor of 91 understated it). Held experimental — hold promotion pending #560.

RESULT (gate, post-training)

v1.4.0-charoffset step-40000, graded bridge-OFF against v0.5.0-bridge.json and bridge-ON against v4.4.0-boundary.json (apples-to-apples). The two are byte-identical on every tag — the bridge is a confirmed NO-OP for this modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (it never fragments po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise., so there's nothing to merge).

• Bridge retirement VALIDATED: us.po_box_real = 90.0 bridge-OFF (floor 89.1). The char-offset format does exactly what it was built for — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learns dotted P.O. BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. intrinsically, so the decode-side spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. bridge can retire at zero cost.
• 16/17 floors pass, several beaten: us.micro 85.4 (▲81.6), localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. 74.1 (▲62.2), regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. 89.5 (▲80.1), unit_real 97, fr.cedex_real 96.7, intersection_real 100, de.native_locality 91.
• One real miss — and bigger than the floor implies: fr.house_number 89.6. The gate FAILs the floor (91) by 1.4pp, but the floor is a conservative bar — v4.4.0 actually measured 97.7, so the true regression is −8.1pp (97.7 → 89.6). Bridge-INDEPENDENT (89.6 both ways), so a genuine char-offset-format regression isolated to FR house_number, not a bridge-off cost. Anchoring to the floor first understated it — the actual-vs-actual read is what matters. Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' held experimental, NOT promoted; hold promotion until the FR house_number cause is understood (#560). The bridge-retirement win is real and independent of this.

What shipped

• First v0.5.0 char-offset trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run is LIVE on A100 (v1.4.0-charoffset, ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. app ap-tG4Os3vGel5MHREGwR9R0X). Step-2000 val: macro_f1=0.627 (streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. 0.81 / house_number 0.99 / localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. 0.70 / regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. 0.65 / postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. 0.70), lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. converging, no NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.. The char-offset format (#519 spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. triple) trains end-to-end — the headline de-risking of the whole v0.5.x line.
• The corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. was completed. The from-source build was BASE-ONLY (11 adapters); the shippable corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. also needs the 7 parity overlays. Re-emitted all 7 (synth-affix/german/countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head./unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise./ po-box-cedexCEDEX (Courrier d'Entreprise à Distribution Exceptionnelle). A French postal routing for high-volume business mail: a CEDEX code delivers directly from a sorting centre, bypassing the local post office. A common negative-space format Mailwoman must parse./intersectionintersectionAn address that names a location by two crossing streets ('5th & Main') rather than a number and street. Mailwoman tags the two streets as intersection_a and intersection_b — a negative-space format that starved the early model. + deepseek-kryptonite, ~485k rows) through the current spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-native aligner so they carry the char-offset triple, merged into the v0.5.0 MANIFEST (689 shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., 676.6M train), re-validated: 18/18 weighted sources present, 0 out-of-bounds spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree., 0 golden-in-val.
• PR #559 — config (v1.4.0-charoffset.yaml), bridge-retirement gate (v0.5.0-bridge.json), align-canonical-shard.mjs, and the train_remote.py reroute (sync_v050 + launcher fix).
• DeepSeek re-align plan drafted (.agents/skills/deepseek-consult/plan-2026-06-12-codepoint-realign.md) for the UTF-16→code-point offset fix (the lasting fix behind the #558 astral-skip stopgap).

What went well

• Verify-before-assert paid off repeatedly. The "overlay gap" alarm was real, but I confirmed the mechanism (loader buckets shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. by parquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. source; an unweighted source's rows train but at the wrong sampling weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.) before crying wolf. The "2 base adapters missing" alarm was a FALSE alarm — they were packed into mixed tail shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., present and trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., same as v4.4.0. Both checks took minutes and prevented wrong conclusions.
• The R2 reroute used the architecture's own grain. Once CLI volume put proved container-blind, the fix was sync_corpus's existing pattern (R2 → container-side rclone), not a bespoke hack.
• Held the GPU. Zero A100 spend until a fresh container provably saw the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. + config. The launcher's own config not found caught the volume issue before any trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. money burned.

What could've gone better

• The overlay gap should have been caught at build time. The rebuild plan's step 5 said "+ the v0.4.x overlay shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. re-emitted natively"; the prior session did the from-source half, validated it, and reported "train-ready" without the overlays. A base-only corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. would have regressed every parity tag and made the bridge-retirement gate untestable. The validation report graded the build in isolation, not against the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. config's source_weights — that cross-check is the fix.
• A long ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour.-infra detour ate most of the shift. Two retries + a marker test + an env-mismatch hypothesis + a container-write test before the cause was nailed. Faster path: the marker test (CLI put → fresh container can't see it) is the 2-minute decisive probe; reach for it first next time.
• Trackio dashboard was down (sister-software/mailwoman-trackio Space not running), so the run logs CSV-only — no live web dashboard for the operator. Should have checked the Space was up before relying on --trackio.

Decisions made autonomously

• Re-emit overlays + complete the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. before launching, rather than launch base-only as a fast format-control signal. Base-only can't ship and can't answer the bridge question (the run's scientific point); the overlay re-emit was bounded (~minutes, builders are spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.-native). Surfaced the gap to the operator; proceeded under "start trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. now" + extended trust once corrected.
• R2 reroute over volume recreation. Recreating the volume would be faster but destroys the container-visible modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' history (every output-* checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.). The R2 path is non-destructive and reusable. Chose it without waiting on the operator since it risks only bandwidth, not data.
• Bridge-retirement gate: inherit v4.4.0 floors verbatim, flag the unpinned thresholds. Rather than fabricate numbers for "over-merge precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1." + "#518 lens", encoded what's contractually pinned and flagged the rest for the operator/DeepSeek. No silent gate drift.
• Enabled --trackio for operator visibility (degrades to CSV-only on failure — which is what happened, harmlessly).

Open questions

1. ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume CLI-write blindness — root cause unknown. CLI volume put writes are invisible to containers on mailwoman-training; container-side writes propagate. Suspected reconciliation stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. from this session's heavy churn (rm -r of 17 old corpora + a 41G put). Will it self-heal? Does it warrant a ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. support ticket or a volume rebuild? (Memory saved: route via R2 meanwhile.)
2. Bridge-retirement gate thresholds. over-merge precision + #518 punctuation lens need numeric floors before the gate is authoritative. Operator/DeepSeek to pin.
3. Will char-offset hold v4.4.0 parity? MOSTLY: 15/17 tags flat-or-better actual-vs-actual, bridge retired at zero cost (po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. intrinsic 90 > v4.4.0 bridged 89.1). The one real casualty is fr.house_number −8.1pp (97.7 → 89.6, #560) — a genuine char-offset regression, NOT noise and NOT the bridge. Lesson re-learned: grade actual-vs-actual, not vs the conservative floor (which said −1.4pp and nearly let an 8pp regression read as trivial).
4. Trackio Space needs waking if a live dashboard is wanted for this and future runs.

Concrete next steps

• When the run finishes (~4h): run the full battery against the final checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. with the v0.5.0-bridge gate (bridge OFF). The decisive read: does us.po_box_real hold ≥89.1 bridge-off? If yes → retire the decode-side spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. bridge. If no → keep it, flip requires_bridge:true for the ship gate, treat as a MISS (don't re-baseline). Compare every tag against v4.4.0 — format-only change should be ~flat.
• Review/merge PR #559 (merge-wall: operator).
• Hand the DeepSeek re-align plan off for the code-point offset fix → corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-v0.5.1 (retires the #558 astral-skip stopgap).
• ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume: decide self-heal vs. rebuild vs. support ticket (open question #1).

Numbers

Shift focus	complete corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. + launch first char-offset trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.
Overlay shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. re-emitted	7 (~485k train rows)
CorpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.	689 shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., 676.6M train / 1.89M val / 1.89M test
Modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' trained	1 (v1.4.0-charoffset, completed 40k steps)
A100 spend before launch	0 (held on the volume issue)
TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. rate	warmed to ~5 steps/s (num_workers:0 loader); ~3.5h total
Gate verdict	FAIL (1 floor: fr.house_number 89.6/91 — but −8.1pp vs v4.4.0 actual 97.7). bridge-retirement PASS (po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. 90 bridge-off, intrinsic > v4.4.0 bridged 89.1)
Bridge	confirmed no-op (bridge-on == bridge-off on every tag) → retire
Infra incidents	1 (ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume CLI-write blindness, both directions; rerouted via R2)
NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol. incidents	0
PRs / issues	PR #559 · issue #560