Performance: what moves parse throughput

In Node, one parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. cannot overlap another in-process. ThreadsthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. are the only lever.

Two levers suggest themselves for making a parser faster: batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. the work, or run it concurrently. Neither one moves this system, and the reasons are invisible from the call site. This page is the measurements, so the next person who reaches for a batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. path finds the receipts first.

Measured 2026-07-16 against the then-shipped v6.3.0 modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (v264-country-softguard, int8), 16-core box. The numbers carry to the current 6.5.0: every release since has been weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.-only on the same int8 graph shape, so throughput is unchanged (parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. latency is graph-bound, not weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.-bound).

Where the time goes

A parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. is 93.2% ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. forward pass. Tokenizing, the anchor/gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture./countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. channel lookups, ViterbiViterbi decodingA dynamic programming algorithm that finds the most likely sequence of hidden states (labels) given a sequence of observations (token emissions). Mailwoman uses Viterbi over a linear-chain CRF to produce globally coherent BIO label sequences from per-token model scores., and building the tree add up to 0.25 ms, under 7% of the wall clock.

If the forward pass were free, parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. would be 14.6x faster. Every lever below is an attempt to collect that 14.6x, and every one of them fails for the same reason: the forward pass is not idle-waiting on anything. ONNX RuntimeONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. already spends 8.02 cores on a single batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=1 inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece., so there is no slack to reclaim.

Concurrency does nothing

1.00x flat, from one worker to sixteen, on both parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and full geocode:

workers	parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. addr/s	scaling
1	274.0	1.00x
2	276.1	1.01x
4	279.4	1.02x
8	274.6	1.00x
16	274.5	1.00x

Two things hold the threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases., and either alone is enough:

• onnxruntime-node's session.run() blocks the JavaScript threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases.. It does not release to the libuv pool. Promise.all over N inferencesinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece. runs them one after another, no matter what N is.
• node:sqlite reads are synchronous by design — the resolution ladder is a sync interface (see ARCHITECTURE). The resolve half of a geocode blocks too.

The giveaway is that the numbers come out identical rather than merely close. Contention would show as a curve; a flat line means nothing overlapped at all.

This is why MAILWOMAN_BATCH_CONCURRENCY was removed on 2026-07-16. It defaulted to 8, it was documented as a tuning knob, and it had never done anything. Setting it to 1 and to 16 produced the same throughput to three digits. POST /v1/batch is a sequential loop now, which is what it had been in practice all along.

The same lever works in Python, where intra_op=1 plus 16 threadsthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. buys 1.64x, because Python's ORT releases the GIL during a run. That conclusion doesn't port. The binding is what differs.

Batching does not pay

The obvious counterexample is deepparse, which parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. at 3.1 ms/addr batched against 32.5 ms sequential, a 10x win. That 10x is amortized per-call Python and torch overhead. We don't carry that overhead, so there's nothing for batching to amortize. Our sequential path (~300 addr/s) already matches deepparse's batched throughput (320 addr/s), and beats its sequential latency 4.7x.

Measured on the shipped int8 graph, re-exported so it could accept a batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. at all:

batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.	ms/addr	addr/s	scaling
1	3.29	304.1	1.00x
4	2.93	341.7	1.12x
8	3.30	303.5	1.00x
32	4.64	215.6	0.71x
64	5.33	187.7	0.62x

It peaks at 1.12x and then goes backwards. fp32fp32 / fp1632-bit and 16-bit floating-point formats. Mailwoman trains in bf16 (a 16-bit variant) and exports the ONNX model in int8 for size. behaves the same way (1.24x at B=4). Batching raises arithmetic intensity, which helps when cores sit idle waiting on memory, and ours don't. Past B=8 the working set stops fitting and you pay for the privilege.

On CPU, don't build a batching path. On GPU or WebGPU the arithmetic-intensity story differs and batching plausibly pays, but nobody has measured it. Treat that as an open question.

A batch axis that isn't

export_onnx.py asks for a dynamic batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. dimension on all eight inputs, and until 2026-07-16 it silently didn't get one. The graph advertised ['batch', 'sequence'] on every input while its outputs were pinned to [1, 'sequence', 33], and any batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. above 1 died inside an attentionattentionThe core mechanism inside a transformer encoder. Each token's representation is updated by looking at every other token, with learned weights deciding how much each one matters. reshape:

Input shape:{64,4,1152}, requested shape:{64,1,3,384}

That 1 is the batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. dimension, traced in as a literal.

The cause is torch.export's 0/1 specialization: a dimension of size 0 or 1 is specialized to a constant unconditionally, even when dynamic_shapes marks it dynamic. The export traced with batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=1 example inputs, so the request was discarded while the input still got labelled 'batch', which is why the metadata lied. Tracing with batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=2 fixes it:

dummy_batch=1  logits_out=[1, 'sequence', 33]        B=4:FAIL
dummy_batch=2  logits_out=['batch', 'sequence', 33]  B=4:OK  B=32:OK

Hence export_to_onnx(..., dummy_batch=2). It defaults to 1, so the shipped artifact is unchanged — the batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=2 re-export reproduces it byte-for-byte at 39.4 MB. The fix is in the tree because an artifact that advertises a capability it doesn't have is a trap for whoever trusts the metadata, not because batching is coming.

Don't try to reproduce the specialization with a small modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. A minimal reshape repro exports fine at batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.=1 on torch 2.13 — it comes out of the transformersneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' attentionattentionThe core mechanism inside a transformer encoder. Each token's representation is updated by looking at every other token, with learned weights deciding how much each one matters. internals, so only the real checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. settles it.

What works

Crossing a threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. boundary is the only thing that moves this.

geocodeStream already does it: an async iterator over a spliterator worker pool, each worker rebuilding its own classifier and lookups, yielding in completion order.

Expect a modest return. GeocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. is memory- and I/O-bound on a shared multi-gigabyte database, and a measured NPPES sweep peaked at two workers (~1.4x), degrading from there (four workers ≈ baseline, six ≈ no gain). Each worker also pays for its own 38 MB session and its own open database. Sweep it for your data and your box rather than reaching for availableParallelism().

Parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.-only threading has no database under it and no contention, so it should scale further than geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. does. Nobody has measured it. Python's intra_op=2 / 8 threadsthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. hit 1.57x, which is a hint about the shape and not a figure to quote.

Before you re-measure

Two traps, both of which produced a wrong answer during the original sweep:

• Warm up, and repeat. A single cold run showed concurrency at 1.19x on the geocode path and came close to buying the knob a reprieve. Repeats showed no trend whatsoever, with c=1 beating c=8 half the time. The spread across identical configs is ~13%, wider than most effects worth chasing.
• Reach the real runner. The forward-pass split needs classifier.cfg.runner. Instrumenting the wrong field reports a confident 0 ms and a clean 0%, a magnitude standing in for its own absence. Same bug class as the meaning of zero.

The probes live in scratchpad/ on the branch that produced this page: batch-export-probe.py, ort-threads-probe.mjs, parse-profile.run.ts, parse-concurrency.run.ts, parse-cpu.run.ts.