Training pipeline

The trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. turns raw address data sources into a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' file that ships on npm. This article walks through each stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). end-to-end. The Corpus construction article digs into the first three stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). in more detail.

The full pipeline

We will walk through each box.

Stage 1 — Adapters

Each raw data source has a TypeScript adapter that reads the source and emits CanonicalRow objects:

interface CanonicalRow {
	raw: string // the address string
	components: Partial<Record<ComponentTag, string>> // the labelled parts
	country: string
	locale: string
	source: string // 'usgov-nppes', 'ban', etc.
	source_id: string
	license: string
}

Adapters live under corpus/src/adapters/. Each one is responsible for:

• Reading its source's native format (CSV, NDJSON, JSON, shapefile).
• Mapping source-specific column names to Mailwoman's component vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it..
• Composing a plausible raw string from the components (when the source does not provide one).
• Stamping the correct licence on every row.

There are currently 14 adapters: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. (admin + postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.), BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure. (France), TIGERTIGERThe US Census Topologically Integrated Geographic Encoding and Referencing database. Used as a corpus source for street-segment data. (US Census), NPPES (US healthcare), HRSA, IMLS, NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses. (US DOT), three stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. notary/contractor sources, and a few others. corpus-v0.3.0 enabled 11 of them, contributing 677 million aligned rows.

Stage 2 — Alignment

The alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. step takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row.

The algorithm:

1. For each (tag, value) in components, find value inside raw. First try a verbatim substring match; if that fails, fall back to fuzzy match via Levenshtein distanceedit distanceThe minimum number of single-character insertions, deletions, or substitutions needed to turn one string into another (Levenshtein distance). Used in corpus alignment and record matching. with a tunable threshold.
2. If any component cannot be located in raw, quarantine the row with a reason like component-not-found:locality. Quarantined rows do not go to trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.; they pile up for later inspection.
3. Tokenize raw. Mark the first tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. in each component spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. as B-tag, subsequent tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. as I-tag, and unaffiliated tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. as O.

The quarantine pile catches data quality issues. corpus-v0.3.0 quarantined 214,118 rows out of 677 million (0.03%) — mostly cases where the source's components.region was a name the source-side raw rendered as an abbreviation, or vice versa.

Stage 3 — Splits

Aligned rows are split into three groups: train (about 90%), val (a few percent, used during trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. to monitor progress), and test (held out for the final evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.).

The split is localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.-aware: rows that share a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. stay in the same split. This prevents leakageleakageTrain/test contamination that inflates reported accuracy when eval data has effectively been seen in training. Mailwoman guards it with held-out-geography evals and locality-aware splits. — if "Brooklyn, NY" appears in trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., no Brooklyn row appears in val or test. Without this, the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' can memorize specific localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. strings and look better on evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. than it is.

The split assignment is written to SPLIT_MANIFEST.json so any later step can reproduce which rows went where.

Stage 4 — Sharding

Once split, rows are written to ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. files at 1 million rows per shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.. ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. is a columnar format that streams well: the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data loader can read row groups one at a time without loading the whole file into memory.

A v0.3.0 build produces about 30 train shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. plus 1 val shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. plus 1 test shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row.. Total size ~30 GB. The intermediate/ directory (per-adapter raw NDJSON dumps, useful for debugging) takes another ~400 GB but gets garbage-collected after the next trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. run.

Stage 5 — Training

TrainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. runs in Python on the lab's GPU (AMD Radeon 780M). The trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. script:

1. Loads the tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. (tokenizer.model).
2. Builds the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' architecture (MailwomanCoarseEncoder + LinearChainCRF).
3. Streams batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. from the train shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. (one ParquetParquetThe open columnar file format the corpus is written and streamed in. The training pipeline reads shards row-by-row from Parquet. row group at a time, 32 examples per batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU., 4 gradient accumulationgradient accumulationSumming gradients over several minibatches before taking an optimizer step, to simulate a larger effective batch size under limited GPU memory. steps → effective batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. 128).
4. Forward pass: encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). produces per-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. emissions → CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. computes negative log-likelihood of gold sequence → lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. is CE(emissions, gold) + 0.05 × CRF_NLL.
5. Backward pass: gradientsgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. flow through CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. + encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). + tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. embeddingsembeddingA vector of numbers representing a token (or other item) so that similar items sit near each other in vector space. The first thing the model does is turn each token into an embedding..
6. OptimizeroptimizerThe component that decides how to update parameters from the gradient — Adam/AdamW being the common choice, adding momentum and per-parameter scaling on top of plain gradient descent. step (AdamWoptimizerThe component that decides how to update parameters from the gradient — Adam/AdamW being the common choice, adding momentum and per-parameter scaling on top of plain gradient descent., learning ratelearning rate (LR). How big a step training takes along the gradient each update. Too high and training diverges; too low and it crawls. Mailwoman warms it up, then decays it on a cosine schedule. 1.5e-4, cosine decaylearning-rate scheduleA plan for changing the learning rate over training — Mailwoman's is linear warmup to a peak, then cosine decay toward zero. Smooths early instability and lets the model settle at the end. with 1,000-step warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup.).
7. Every 250 steps: evaluate on the val shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., log val lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. + macro F1macro F1The unweighted average of per-class F1 scores — treats every class equally. Mailwoman's primary label-level eval metric..
8. Every 100 steps: save a full checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. (modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' + optimizeroptimizerThe component that decides how to update parameters from the gradient — Adam/AdamW being the common choice, adding momentum and per-parameter scaling on top of plain gradient descent. + scheduler + RNG stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.).

Save-every-100 is a defensive measure for the lab's GPU: it has a firmware quirk that fires a GPU Hang exception every 30–60 minutes under sustained load. The train_with_resume.ts wrapper auto-restarts on hang and resumes from the latest checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.. With this in place, trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. is robust to multiple hangs per session.

The v3.0.0 run trained for 1,800 steps (out of a planned 50,000) before the val metric peaked and started degrading. We early-stopped at 1,800 to ship the best checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang..

Stage 6 — Eval

After trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). runs the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' against the held-out golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals. (data/eval/golden/v0.1.2/, 4,535 hand-labelled entries). For each entry:

• Tokenize the input.
• Run the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (with CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores. decode).
• Decode the BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components. back into component strings (find the contiguous spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree., slice out the original characters).
• Compare to the gold components.

The output is a markdown report and a JSON dump under docs/articles/evals/. Per-component precision, recall, F1, plus an overall "exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness." rate and a calibration histogram (does confidence match accuracy?).

The evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. report is committed to git as part of the ship PR — see the latest Tier 2 ship's eval report for the v3.0.0 numbers (filename preserves the historical "stage2" naming).

Stage 7 — Export to ONNX

PyTorch's trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. graph is not directly portable to JavaScript or other runtimes. ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. (Open Neural Network Exchange) is a standardized modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' format that runs in many runtimes (onnxruntime-node, onnxruntime-web, ONNXRuntimeONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. for mobile, etc.). The export step:

• Traces the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s forward pass with a sample batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU..
• Records each operation as an ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. node.
• Writes the result to model-stage2-step-XXXXXX-fp32.onnx.

A parity check then runs the same inputs through the original PyTorch modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' and the exported ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' and asserts the outputs agree to within 1e-4. Mismatches mean the export silently dropped or transformed an operation. v3.0.0's parity check showed max difference of 1.4e-5 — well within tolerance.

Stage 8 — int8 quantization

ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' start in fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. (32-bit floating point). For shipping, Mailwoman runs dynamic int8 quantizationint8 quantizationA model compression technique that stores weights as 8-bit integers instead of 32-bit floats, reducing model size (~4×) with minimal accuracy loss. Mailwoman ships int8-quantized ONNX models (~30 MB) for fast loading in browsers.: each layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6.'s weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. get converted to 8-bit integers at load time. This:

• Cuts the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' file from 37 MB to 25 MB.
• Speeds up inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. on CPU (smaller arithmetic).
• Costs about 1–2% accuracy on the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. — measurable but acceptable.

Quantization is a built-in onnxruntime.quantization operation, applied once at ship time.

Stage 9 — Package

The final step packages the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' into the npm-publishable format:

packages/neural-weights-en-us/
├── model.onnx           # 25 MB int8 model
├── tokenizer.model      # 470 KB SentencePiece
├── model-card.json      # eval numbers + hardware metadata
├── README.md            # short usage doc
└── package.json

The model-card.json is the canonical record of what shipped: which corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. version, which checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang., what hparams, what evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. metrics. evals/scores-by-version.json mirrors the same info but is keyed by run ID for cross-version comparison.

Both packages (en-us, fr-fr) get bumped to a new npm version (v3.0.0 for the TiertierInternal versioning of which label classes the model emits. Tier 1 is the coarse components (country, region, locality, postcode); Tier 2 adds venue, street, house_number; Tier 3 (future) would add attention, po_box, and POI venue subtyping. Historically called 'Stage 1/2/3' before the runtime-pipeline naming made that ambiguous. 2 ship) and published.

See also

• Corpus construction — StagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 1–4 in more detail
• How the model reasons — what the trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' actually does with a tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.
• ONNX runtime — what happens after ship