4 docs tagged with "tokenizer"

This article describes where Mailwoman is heading. The work is tracked in GitHub issues and the plan/ directory. Status is current as of May 2026.

Date: 2026-05-29

SP UDS are stored as the post-normalization form. SP normalizes ASCII spaces to U+2581 (▁) before matching against the UDS vocab, but the user-supplied string keeps the raw space.

Tokenization is the step that turns an input string into a list of small pieces called tokens. Every downstream component — rule classifiers, the neural classifier, the solver, the resolver — works with tokens, not with raw strings.