v3.0.0-span-head 2k probe — the head never trained; the gate does not adjudicate

Gate (config-canonical, quoted verbatim from v3.0.0-span-head.yaml):

seg@1 > token@1 on the parity corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. Baselines (v264, ship config): streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 0.573; a segmentsegmentA punctuation-bounded chunk of the normalized input — the comma-separated parts of 'Portland, OR' — used to give downstream stages structural context. decode over the SUMMED-BIO stand-in scored 0.453. Secondary read: oracle@10 must RISE from 0.749. If seg@1 < tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1: do NOT tune span_loss_weight and re-run — that is the treadmill. One diagnostic (is the lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. decreasing?), then fork to the operator.

Result: the gate reads FAIL, but it does not adjudicate the arc. Per the pre-registered diagnostic, this is under-trained, not falsified — and the honest call is the fork, which is why this doc exists rather than a relaunch.

What ran

Run	ap-2xQYXyXtcLhyIsdoSpTp0v, A100, 2000/2000 steps, ~31 min
Config	v3.0.0-span-head.yaml — ONE variable vs v264: use_span_scorer: true, span_loss_weight: 0.5
Init	init_from v264 step-008000 — missing=9 unexpected=0 (exactly the 9 new spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. tensors)
NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.	none. fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. partition math held where the v0.5.0 bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.'d twice
New params	101,076 of 39,360,131 (0.26%)

The numbers

Channel-starvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). (see the caveat below) — relative comparison only:

tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 (BIO argmax)	93/267 = 0.348
seg@1 (semi-Markov argmax)	1/267 = 0.004

Why this is under-training, not falsification

The pre-registered diagnostic is "is the lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. even decreasing?". It is, and it is nowhere near converged:

step	50	450	850	1200	1600	2000
train_loss	26.41	25.33	21.22	18.84	18.12	17.77

• Still falling at 2000 (18.12 → 17.77 over the last 400 steps). Not a plateau.
• The raw spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. NLLNLL (Negative Log Likelihood). The standard form of a loss function in probability-based models: loss = −log P(correct label | input). The CRF uses the NLL of the entire label sequence. is ~35 (17.77 ÷ the 0.5 weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., less the ~1 CE). A converged semi-CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. sits at O(1). The headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. is a small fraction of the way in.
• The decode looks random, exactly as a near-random headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. should: Korunni 810, Praha → ▁Kor:street un:region ni:subregion ▁8|1|0:postcode — a type per tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. with no structure. Compare the BIO headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. on the same input: ▁Kor|un|ni:street ▁8|1|0:house_number ▁Pra|ha:locality — correct.

The probe was mis-specified, and that is my error. lr: 1e-5 was inherited verbatim from v2.6.4 — a recipe that fine-tunes existing weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.. The spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. is randomly initialized. A fresh headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. at a fine-tuning LR for 2k steps cannot train, and the one-variable discipline (correctly) stopped me from also changing the LR, so the probe tested "does a barely-initialized headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. beat a fully-trained one" — a question with a known answer.

The BIO head is untouched — the byte-identity property held in a real run

The step-2000 evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error., which is the useful result here:

	streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.	house_number	postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.	localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.	regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.	macro
v300 @ 2k	0.869	0.995	0.999	0.765	0.754	0.7405

The spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. trained alongside without perturbing the BIO path — the unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. test's invariant, now confirmed under real gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping.. That much of PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1 is proven.

Caveat that limits this harness (a second design error, also mine)

eval_seg_at_1.py runs Python-side against the torch checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang., deliberately, to avoid depending on the PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary).-2 ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. export. But the Python side feeds no anchor/gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture./countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. channels, no postcodeRepair, no word-consistency heal — the #718 channel-starvation trap. So its tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 reads 0.348 where the JS harness reads 0.573 on the same modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. The absolute numbers here are NOT comparable to mailwoman eval parity --weights-cache, and the script now prints that in its output.

The relative gate survives (both headsattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. read the same starvedstarvedA tag with too little training representation to learn — near-zero F1 — because the corpus adapter never emits examples of it (intersection tags sat at 0% until an intersection synthesizer existed). encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.), but any future PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary).-1 re-run should either feed the channels or accept that only the comparison is meaningful.

Two harness bugs were found and fixed getting here, both by refusing to accept a tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 that couldn't reproduce a known baseline:

1. from_pretrained() never passed map_location, so a GPU-trained checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. could not load on a CPU-only box at all. This affects every local grading run, not just this one.
2. The gate concatenated streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-family piecesECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated)., dropping the O-labelled bare ▁ separator and welding words (▁5|th|▁|Ave → "5thAve"). Now slices by char offsets. tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.@1 0.285 → 0.348.

The fork (operator's call — the pre-registration says so)

The arc is not falsified. The question the probe was meant to answer is still open. Options:

1. Re-probe with a headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-appropriate LR — e.g. a param-group LR for span_scorer/semi_crf at 1e-3 while the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). stays at 1e-5. This is standard for a fresh headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. on a pretrained encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters)., and it is not the treadmill (the treadmill is opposite-direction knob oscillation; this is fixing a mis-specified probe). Cheapest real test of the hypothesis.
2. 8k at the current LR — tests whether duration alone gets there. Slower, and the trajectory suggests it would still be far from converged.
3. Park Phase 1 and revisit with the option-C recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. channel bundled (violates one-variable, but the 66% bare-fragment class is the bigger prize anyway).

My recommendation is (1): it is one variable (the headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.'s LR), it costs ~30 min of A100, and it is the only option that actually tests the architecture rather than re-testing a random headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads..

Artifacts

• Branch feat/727-span-head — 8 commits, 21 unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. tests (both DP routines brute-force verified).
• CheckpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.: output-v300-span-head-s42/checkpoints/step-002000 on the ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. volume.
• train_log.csv pulled to /tmp/v300-log.csv.