ONNX runtime

ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. (Open Neural Network Exchange) is a standardized format for serializing trained neural networksneural networkA model made of layers of simple numeric units whose connection strengths (weights) are learned from data. The transformer encoder at Mailwoman's core is a neural network.. ONNX RuntimeONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. is the family of inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. engines that consume those files. Mailwoman uses ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. so the same trained modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' can run in Node.js, browsers, mobile devices, or anywhere else with an ONNX RuntimeONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. build.

Why ONNX and not PyTorch directly

PyTorch is great for trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.. It is heavy for inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece.. A PyTorch install is ~700 MB of Python wheels and depends on a specific Python version, CUDA toolkit, and C++ runtime. Shipping that to a browser is not feasible. Shipping it to a serverless function is painful.

ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. is the lingua franca that decouples trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. framework from inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. runtime. Mailwoman trains in PyTorch (because that is the easy path for the team), exports to ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. (because that is the portable format), and ships ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. files. Consumers pick the runtime that fits their environment:

• onnxruntime-node — Node.js bindings. Used by @mailwoman/neural (the default Node SDK).
• onnxruntime-web — WebAssembly + optional WebGPU. Used by @mailwoman/neural-web (the browser SDK).
• onnxruntime-mobile — for iOS / Android (not currently shipped but architecturally supported).

The same model.onnx file works in all three.

How the model file is structured

An ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is a graph of operations: matrix multiplies, attentionattentionThe core mechanism inside a transformer encoder. Each token's representation is updated by looking at every other token, with learned weights deciding how much each one matters. computations, activations, layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. normalisation, etc. Each operation has typed inputs and outputs. The graph is serialized as a Protocol Buffer.

For Mailwoman's encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters)., the export traces this kind of structure:

inputs:
  input_ids:        int64[batch, seq]
  attention_mask:   int64[batch, seq]

graph:
  embedding → position_add → layernorm → dropout
  block_1 (attention + ln + ffn + ln)
  block_2 (...)
  ...
  block_6
  final_layernorm
  classifier (linear: 256 → 21)

outputs:
  logits:           float32[batch, seq, 21]

The CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality. is not part of the ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. graph in v3.0.0. The ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores. decode runs in JavaScript (or will, post-v0.4.0). The transition matrixtransition matrixThe CRF's learned table of per-label-pair scores. It encodes which BIO transitions are preferred or forbidden — e.g. that an I-tag must follow a matching B- or I-. gets exported as a separate tensor in the ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. file but the algorithmic loop is not.

Inference in Node.js

Loading and running the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' on the server is straightforward:

import { InferenceSession } from "onnxruntime-node"
import { readFileSync } from "node:fs"

const session = await InferenceSession.create(readFileSync("model.onnx"))

const inputIds = new BigInt64Array([/* tokenized address */])
const attentionMask = new BigInt64Array([/* 1s for real tokens, 0s for padding */])

const results = await session.run({
	input_ids: new ort.Tensor("int64", inputIds, [1, seq_len]),
	attention_mask: new ort.Tensor("int64", attentionMask, [1, seq_len]),
})

const logits = results.logits // float32[1, seq_len, 21]
const labels = argmax(logits) // pick best label per token

The whole inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. path — tokenize, run modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', decode labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. — takes ~5 ms on a modest CPU for a typical address. The session-creation step (loading the file, JIT-compiling the graph) takes ~30 ms and is amortized across many calls.

Inference in the browser

The browser path uses onnxruntime-web, which is the same engine compiled to WebAssembly. Loading is a bit more involved because the WASM blob is several MB:

import * as ort from "onnxruntime-web"

const session = await ort.InferenceSession.create("/mailwoman/model.onnx", {
	executionProviders: ["wasm"], // or ["webgpu", "wasm"] for GPU acceleration
})

// Same API as the Node version
const results = await session.run({ input_ids, attention_mask })

The browser path optionally uses WebGPU (where available) for GPU acceleration. WebGPU is enabled by default in modern Chrome and Edge; Firefox and Safari are catching up. When WebGPU is not available, the runtime falls back to WASM SIMD, which is roughly 2–3x slower but still fast enough for interactive use.

@mailwoman/neural-web wraps this with the same API as @mailwoman/neural so consumer code rarely has to know whether it is running in Node or in the browser.

Size budget

ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. gives us portability. int8 quantizationint8 quantizationA model compression technique that stores weights as 8-bit integers instead of 32-bit floats, reducing model size (~4×) with minimal accuracy loss. Mailwoman ships int8-quantized ONNX models (~30 MB) for fast loading in browsers. gives us a small enough file to ship to the browser.

representationhidden stateThe model's internal vector for a token after the encoder has mixed in surrounding context. The contextualized representation the classifier head reads to assign a label.	modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' file size
PyTorch checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. (fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size.)	37 MB
ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. export	37 MB
ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. int8 quantized	25 MB

25 MB is the number Mailwoman ships. Combined with the WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. SQLite slim distribution (~35 MB), the total cold-load for the browser demo is about 60 MB. The browser caches everything on first load; subsequent visits are instant.

For context, a single high-resolution photo on a typical news site is 200 KB to 2 MB. A modern app bundle is 5–20 MB. 60 MB is large but not unreasonable for a one-time cold load.

Export caveats

A few subtle issues come up during ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. export:

• Dynamic axes. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' needs to handle variable batch sizesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. and variable sequence lengths. Both are declared as "dynamic axes" during export. The exported modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' graph contains shape inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. logic so the runtime can adapt at call time.
• Operations that have no ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. equivalent. Most PyTorch ops do. A few (some custom ROCm kernels, some experimental quantization ops) do not. Mailwoman avoids them.
• The dynamo exporter path. PyTorch 2.x ships two ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. exporters: the legacy TorchScript-based one and the newer dynamo-based one. The legacy path crashes on transformers ≥ 5's attentionattentionThe core mechanism inside a transformer encoder. Each token's representation is updated by looking at every other token, with learned weights deciding how much each one matters. masking implementation; we use the dynamo path with onnxscript post-processing.
• CPU-only export on this GPU. ROCm + bf16bf16 (bfloat16). A 16-bit floating-point format used during training. Half the memory of fp32 with similar numeric range, which lets the training batch fit on the lab's GPU. + ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. tracing hangs on the lab's hardware. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is moved to CPU before the export call. This is slow (a few minutes) but only runs at ship time, not per-iteration.

Where this lives in the code

• Export script: corpus-python/src/mailwoman_train/export_onnx.py
• Quantization: corpus-python/src/mailwoman_train/quantize.py
• Node runtime wrapper: neural/onnx-runner.ts
• Web runtime wrapper: neural-web/web-onnx-runner.ts
• Demo integration: docs/src/pages/demo/index.tsx

See also

• How the model reasons — the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). computation the ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. graph exports
• Training pipeline — where the ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. file comes from
• CRF decoder — the part that is not in the ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. graph (yet)