Skip to main content

The pipeline contract

You don't have to take Mailwoman's pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. as-is. The runtime coordinator (createRuntimePipeline) accepts each pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). as an injectable function or interface; an integrator can swap any of them for a custom implementation without forking the core.

This article is the practical "how do I plug in" companion to:

  • The staged pipeline — the narrative for why the stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). exist
  • STAGES.md (reference) — the full per-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). contract with type definitions, error semantics, and edge cases

Read those first if you're new. Use this page when you already know the shape and want to ship a custom stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone)..

Stage signatures

import type { RuntimePipelineStages } from "@mailwoman/core/pipeline"

interface RuntimePipelineStages {
normalize?: (raw: string, opts?: { locale?: string }) => NormalizedInputLite
computeQueryShape?: (input: NormalizedInputLite | string, opts?: { locale?: string }) => QueryShapeLite
detectLocale?: (input: NormalizedInputLite, shape: QueryShapeLite, opts?: { hint?: LocaleTag }) => Promise<LocaleHint>
classifyKind?: (input: NormalizedInputLite, shape: QueryShapeLite, locale: LocaleHint) => Promise<QueryKindResult>
// Coarse country router (#244). Soft prior for the resolver's re-rank, never the classifier;
// off in the bare core coordinator (byte-stable), wired on by default in the `mailwoman` package.
placeCountry?: (normalizedText: string) => { country: string | null; confidence: number; posterior?: Record&lt;string, number&gt; }
groupPhrases?: (input: NormalizedInputLite, shape: QueryShapeLite, locale: LocaleHint) => Promise&lt;PhraseProposal[]&gt;
classifier?: { parse(text: string, opts?: ClassifierOpts): Promise&lt;AddressTree&gt; }
fst?: FSTMatcherLike // FST gazetteer matcher — produces emission biases during classification
resolver?: { resolveTree(tree: AddressTree, opts?: ResolveOpts): Promise&lt;AddressTree&gt; }
}

Every stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). is optional. When you omit one, the coordinator either substitutes a no-op stub (normalize, computeQueryShape, detectLocale, classifyKind) or skips the stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). entirely (classifier, placeCountry, resolver).

Decode-time constraint layers (v4.3.0+)

The neural classifierneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' carries two constraint layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. configured at construction, not as pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). — they run inside classifier.parse and are part of the SHIP CONFIG contract (the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' card's notes name the required settings per release):

  • Conventions maskconventions maskA decode-time constraint layer keyed by the model's own address-system detection (the exported locale head): tags that are ungrammatical in the detected system are removed from the Viterbi vocabulary, and the system's postcode shape arms a snap-only repair pass. The first slice forbids USPS street-affix decomposition for French. Same knowledge-outside-the-weights property as the gazetteer anchor — add a codex conventions row, no retrain. (addressSystemConventions: "auto" | SystemCode, v4.3.0+): the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s exported localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. detects the address system; the codex conventions row removes that system's ungrammatical tags from the ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores. vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. and arms the postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one.-shape snap repair. Threshold-gated (0.8) — never fires on a guess; undetected systemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. unmasked.
  • SpanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. bridge (bridgePunctuationGaps: true, REQUIRED from v4.4.0): merges same-tag fragments split at unlabeled intra-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. punctuation ("P.O. BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise." decoding as P+O+Box) — the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. format cannot express punctuation inside spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree., so the bridge is decode-side containment. Separator punctuation (commas) never bridges; space-only gaps never bridge.

Running a v4.4.0+ modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' WITHOUT these settings reproduces measured gate failures (po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. 60.4 vs 89.1 with the bridge; the FR digit-split class without conventions) — they are not optional enhancements for those weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., they are the decode contract.

Replacing a stage

import { createRuntimePipeline } from "mailwoman"

const myLocaleDetector = async (input, shape, opts) => {
if (opts?.hint) return { locale: opts.hint, confidence: 1.0, alternatives: [], source: "caller" }
// Your detection logic here — fastText, a CLD, an LLM, whatever.
const guess = await myModel.classify(input.normalized)
return { locale: guess.tag, confidence: guess.score, alternatives: [], source: "detected" }
}

const pipeline = createRuntimePipeline({
detectLocale: myLocaleDetector,
// Other stages default to the shipped implementations.
})

const result = await pipeline("8 rue Lafayette, Paris")

Same pattern for any stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).. The defaults you don't override (e.g. @mailwoman/normalize, @mailwoman/query-shape, @mailwoman/kind-classifier) keep running.

Error semantics

Not every stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). failure is treated the same. The coordinator distinguishes graceful stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). (failure produces a degraded but valid result) from non-graceful stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). (failure surfaces to the caller).

StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).If it throws
normalizePropagates. Input preprocessing is a contract; a crash is a bug.
computeQueryShapePropagates. Same rationale.
detectLocalePropagates. A localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. detector that crashes signals a real fault, not noise.
classifyKindPropagates. Kind classification runs on the QueryShape we just computed — there's nothing external that can poison it.
classifier.parseSwallowed. Returns an empty AddressTree; pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. continues. The classifier runs against external weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. / arbitrary user input; defensive.
resolver.resolveTreeSwallowed. Returns the classifier's tree unchanged. Backend may be unavailable; we surface what we have rather than fail the whole call.

This asymmetry is intentional. StagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 1, 2, 2.5 are pure functions over the input; if they crash, something is wrong with the request shape and the caller needs to know. StagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 3 and 6 are wrapped because their dependencies (ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', SQLite database, network) are points where production failures legitimately happen and degrading gracefully beats taking the whole query down.

Cancellation

PipelineOpts.signal: AbortSignal is honored between stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone)., not within them.

const controller = new AbortController()
setTimeout(() => controller.abort(), 100)

try {
const result = await pipeline("…", { signal: controller.signal })
} catch (err) {
// err.name === "AbortError" if the coordinator caught the signal between stages.
// err is whatever you passed to controller.abort(reason) if you supplied a reason.
}

If you abort while a stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). is mid-execution, that stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). runs to completion before the abort takes effect. The longest cancellation latency is one stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).'s runtime — typically the neural classifierneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (~10-70ms p99 for the en-US weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.). Fine-grained mid-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). cancellation is a future enhancement that requires plumbing signal into each stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).'s contract.

Aborting before any stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). runs throws immediately. Aborting between stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). skips the rest of the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package..

Timing budget

PipelineResult.timing records per-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). wall time in milliseconds. The keys present depend on the path the coordinator took:

KeyAlways presentNotes
normalizeyes
query-shapeyes
locale-gateyes
kind-classifieryes
token-classifyfull path only, when classifier wiredAbsent on fast-path; absent when no classifier injected
resolvewhen resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. wiredPresent on both full and fast-path

The path field tells you which branch ran: "full" (stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 3-5 ran) or "fast-path" (kind classifierkind classifierStage 2.5 of the runtime pipeline: categorizes the input into one of seven query kinds (structured_address, postcode_only, locality_only, intersection, po_box, landmark, vague) so the coordinator can route to the right parsing strategy. + QueryShape agreed on a trivial input, stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 3-5 skipped, tree built from QueryShape alone).

Use mailwoman parse --benchmark <N> for percentile breakdowns over many iterations against a real input.

Fast-path criteria

The coordinator short-circuits stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 3-5 when:

  1. forceFullPipeline is not set on PipelineOpts, AND
  2. classifyKind returned confidence ≥ 0.95, AND
  3. The kind matches a known shape signal:
    • postcode_only → QueryShape has a postcodepostcodeThe country-specific postal code (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one. knownFormats hit
    • locality_onlytotalLength ≤ 30 AND characterClass === "alpha"

The fast-path tree is built from QueryShape's format hit alone — useful even without a resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. wired (a consumer who just wants the parsed structure for "10118" shouldn't pay for the classifier).

When to swap a stage

Common reasons:

  • Custom localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. detector — replace detectLocale with a fastText / cld3 / commercial detector when you have a specific traffic profile
  • In-process resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. — swap resolver for an embedded WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. SQLite or in-memory gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.
  • Different classifier — your own ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', a rule-based classifierrule-based classifierMailwoman's legacy v0 parser — a library of deterministic token classifiers (house number, street suffix, postcode, place name, etc.) composed by priority. Now primarily used for corpus labeling, fallback classification, and arbitration diagnostics., a remote inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. endpoint
  • Test fakes — every stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). takes a vi.fn() / sinon stub directly; no special harness needed

You should not swap a stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). to bypass it cheaply. If you want the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. without localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. detection, pass { locale: "en-US" } on every call — the default detectLocale is already a sub-microsecond caller-trust stub. The performance you'd reclaim by swapping is rounding error against the neural classifierneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s milliseconds.

See it in action

Loading demo embed…

Expand the Classification origin section to see which pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). contributed to each parsed component.

See also

  • STAGES.md — the full per-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). type contract
  • The staged pipeline — why these stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). exist
  • QUERY_SHAPE.md — the structural-prior sub-system feeding stagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2 and 2.5