SentencePiece UDS literals with ASCII space silently fail to match

SP UDS are stored as the post-normalization form. SP normalizesnormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input. ASCII spaces to U+2581 (▁) before matching against the UDS vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it., but the user-supplied string keeps the raw space.

Symptom: pass user_defined_symbols=['PO Box', 'SW1A 1AA'] to spm.SentencePieceTrainer.train. The trainer ACCEPTS the list (logs a warning about escaped chars in tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses..vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it., but doesn't fail). The output piece stream never emits 'PO BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise.' as a single piece — instead 'PO' decomposes to byte-fallback piecesECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated). (<0x50> <0x4F>) and the box decomposes into 'B' + byte-fallback chars.

Fix: substitute → ▁ (U+2581 LOWER ONE EIGHTH BLOCK) in every UDS literal before passing to the trainer. Then the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). matches: input 'Send to PO BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. 1234' tokenizes to [..., '▁', 'PO▁Box', '▁', '1', '2', '3', '4'].

Where this lives: corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-python/src/mailwoman_train/tokenizer_train.py — _normalize_uds_for_sp. Test: corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-python/tests/mailwoman_train/test_tokenizer_train.py::test_user_defined_postcode_kept_whole.

Why it bites: there's no error log. SP silently treats the UDS as never-matching, and byte-fallback eats the result. If you're measuring byte-fallback rate (or pulling postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. UDS for must-keep-whole guarantees) and the rate looks higher than expected, check this first.

See also

• Tokenizer A0 — baseline — the tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. retrain that hit this gotcha