4 docs tagged with "Tokenization"

This is a roadmap snapshot, current as of May 2026. For current state, see the scope declaration, How Mailwoman parses an address, How Mailwoman resolves a place, and How it works now.

Date: 2026-05-29

SP UDS are stored as the post-normalization form. SP normalizes ASCII spaces to U+2581 (▁) before matching against the UDS vocab, but the user-supplied string keeps the raw space.

Tokenization is the step that turns an input string into a list of small pieces called tokens. Every downstream component — rule classifiers, the neural classifier, the solver, the resolver — works with tokens, not with raw strings.