fp32-CRF diagnostic — 2026-05-28

The 2026-05-28 night-2 postmortem left one open question after the v0.6.0 NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol. incidents:

CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. on a 33×33 transition table in bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.'d twice. Fix was to disable CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. entirely. The bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. hypothesis remains unconfirmed — needs a fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. follow-up.

Two divergencesdivergenceA training failure where loss descends through warmup, plateaus low for a while, then climbs catastrophically back to its starting magnitude — the model unlearns everything despite no obvious component failure. had been observed:

Attempt	LR	crf_loss_weight	NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol. at step
1	1.5e-4	0.5	~950
2	1.0e-4	0.1	~1700

Both were post-warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup., after the LR reached its constant value. The transition table is 33×33 in StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 3 (vs 21×21 in StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2) with masked -inf entries enforcing the structural BIO grammar (no I-X after O, no I-X after I-Y for X≠Y, etc.). The hypothesis: bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU.'s 7-bit mantissa loses too much precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. against -inf-sentinel arithmetic in the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.'s logsumexp forward pass.

DeepSeek's turn 7 sign-off explicitly required this experiment as a precondition for v0.6.2 — independent of v0.6.2's corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. changes (different variable).

Implementation

crf_fp32: bool = False flag added to ModelConfig (corpus-python/src/mailwoman_train/config.py) and threaded through MailwomanCoarseEncoder (model.py). When set, the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. forward call runs under torch.autocast(device_type=..., enabled=False) with emissions and the mask upcast to fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.:

if self.crf_fp32:
    device_type = logits.device.type
    with torch.autocast(device_type=device_type, enabled=False):
        emissions_fp32 = logits.float()
        crf_mask = attention_mask.to(emissions_fp32.dtype)
        crf_loss = self.crf(
            emissions=emissions_fp32,
            tags=labels.clamp(min=0),
            mask=crf_mask,
            reduction=crf_reduction,
        )

Everything outside the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. call continues in bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. — encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). forward, embeddingembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding. lookup, CE losscross-entropyThe standard classification loss: it penalizes a model for putting low probability on the correct label. Per-token negative log-likelihood is the cross-entropy of each token's label., optimizeroptimizerThe component that decides how to update parameters from the gradient — Adam/AdamW being the common choice, adding momentum and per-parameter scaling on top of plain gradient descent. step. Only the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.'s logsumexp over the transition table runs in fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.. The crf_loss is downcast back to ce_loss's dtype before the dual-lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. sum so the optimizeroptimizerThe component that decides how to update parameters from the gradient — Adam/AdamW being the common choice, adding momentum and per-parameter scaling on top of plain gradient descent. sees one consistent tensor.

Default is False — every existing config remains bit-identical to its prior runs.

Diagnostic config

configs/v0_6_2-crf-fp32-diagnostic.yaml. Identical to v0.6.1-stage3-streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. EXCEPT:

Field	v0.6.1	Diagnostic
crf_loss_weight	0.0	0.5
crf_fp32	n/a	true
max_steps	100000	3000
eval_every_steps	2000	500

3000 steps clears both observed NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol. windows (step 950 + step 1700) with margin. ~30 min on A100.

Result: PASS

TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. reached step 3000 with no NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.. Lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. curve:

Step	train_loss	val_loss	macro_f1
25	43.81	—	—
100	32.21	—	—
500	(warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup.)	—	—
950	1.5647	—	—
1000	1.5375	4.3418	0.2235
1500	0.9044	3.8535	0.2589
1700	0.8934	—	—
2000	0.6809	(next evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.)	—
2500	0.5927	3.6348	0.2979
2825	0.4913	—	—

Both previously-NaNNaN (not a number). A floating-point result for an undefined operation (log of a negative, 0/0). Appearing in the training loss usually halts the run; recovering from it follows the NaN protocol.'d steps (bold) cleared cleanly. macro_f1 climbing monotonically: 0.2235 → 0.2589 → 0.2979. val_loss dropping: 4.34 → 3.85 → 3.63.

The bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. + masked -inf transition hypothesis is empirically confirmed. fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. inside the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.'s logsumexp forward eliminates the precisionprecisionOf the spans the model labeled as a given tag, the fraction it got right. High precision means few false positives. Paired with recall to compute F1. lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.; the rest of the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' runs in bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. for throughput as before.

Implications

1. v0.6.3 can re-enable CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality.. Set crf_loss_weight: 0.5 (or higher) + crf_fp32: true in the v0.6.3 config. The learned transition table that fell out of v0.6.0's failed attempts is now within reach.

2. v0.6.2 stays CE-only. Per the NaN protocolNaN protocolThe NaN-recovery discipline: stop, change one variable, retry, document the hypothesis — never adjust two knobs at once during recovery, or you can't tell which change fixed it.'s "one variable at a time" rule: v0.6.2 already changes corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. composition (synth-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. + new synth-no-streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.). Adding CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. activation in the same release confounds attribution. Defer to v0.6.3.

3. fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.-CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. throughput cost is minimal. The diagnostic ran at ~6 steps/s on an A100-SXM4-40GB, comparable to v0.6.0's CE-only throughput. The autocast boundary only affects the CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. forward (a small linear-in-seq-len operation), not the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters)..

4. The NaN protocolNaN protocolThe NaN-recovery discipline: stop, change one variable, retry, document the hypothesis — never adjust two knobs at once during recovery, or you can't tell which change fixed it.'s "diagnose ONE knob → retry" rule paid off here. The fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. experiment was a 30-min run that closed a question that had been open since the postmortem. The cost of NOT running it was indefinite future uncertainty.

Reproducing

# Push the changed model.py + config.py + new diagnostic yaml to the Modal volume:
modal volume put mailwoman-training corpus-python/src/mailwoman_train/model.py \
  corpus-python/src/mailwoman_train/model.py --force
modal volume put mailwoman-training corpus-python/src/mailwoman_train/config.py \
  corpus-python/src/mailwoman_train/config.py --force
modal volume put mailwoman-training \
  corpus-python/src/mailwoman_train/configs/v0_6_2-crf-fp32-diagnostic.yaml \
  corpus-python/src/mailwoman_train/configs/v0_6_2-crf-fp32-diagnostic.yaml --force

# Clear pyc cache (Modal's documented gotcha — labels.py / model.py changes need this):
modal volume rm mailwoman-training corpus-python/src/mailwoman_train/__pycache__ -r

# Launch the experiment (~30 min on A100):
modal run -d scripts/modal/train_remote.py --config v0_6_2-crf-fp32-diagnostic.yaml --resume none

# Watch:
modal app logs <app-id>  # look for loss=nan in the step 900-2000 range

See also

• 2026-05-28 night-2 postmortem — the open question this diagnostic closes
• v0.6.2 training config — the mainline retrain (CE-only per NaN protocolNaN protocolThe NaN-recovery discipline: stop, change one variable, retry, document the hypothesis — never adjust two knobs at once during recovery, or you can't tell which change fixed it.; CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. deferred to v0.6.3 with these fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. changes)
• Street-supplement architecture — the v0.6.2 retrain context