Pre-publish 2D eval gate

Mechanical guard that runs before any neural-weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. release lands on the default HF channel. Compares a candidate modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s per-tag evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. output against a baseline (typically the current default), applies a two-dimensional threshold, and exits non-zero if any tag violates either dimension.

The shape of the gate is the lesson the 2026-05-28 night-2 postmortem and the Layer 1 eval both pointed at: a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.-only gate would have passed v0.6.1 (no tag regressed > 2pp in recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. on tags where the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' already had > 10% baseline), letting the 1066 dependent_locality hallucinations through. The two-dimensional gate catches both halves.

The 2D rule

FAIL if  (recall drop > recall_threshold_pp  AND  baseline_recall > recall_min_baseline_pct)
      OR (hallucination spike > hall_abs_threshold
          AND new_hallucination_rate > hall_rate_threshold_pct)

Defaults:

Threshold	Default	Why
recall_threshold_pp	2	A drop smaller than 2pp is noise on aggregated golden-set runs
recall_min_baseline_pct	10	Drops on already-low-recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. tags don't move user-visible quality much
hall_abs_threshold	100	Below this is plausibly normal trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. drift
hall_rate_threshold_pct	20	A tag hallucinating more than 20% of its expected count is a structural failure

Each dimension stands alone but they compose: a regression has to clear BOTH gates to ship.

Why both dimensions matter — the v0.6.1 retroactive check

Running the gate against v0.6.0 → v0.6.1 produces:

GATE FAILED: 3 violation(s).
- locality (recall) — recall 39.7% → 31.1% (Δ -8.6pp; baseline > 10%)
- house_number (recall) — recall 79.0% → 75.9% (Δ -3.1pp; baseline > 10%)
- dependent_locality (hallucination) — hallucinated 0 → 1066 (Δ +1066; new rate 2665.0%)

The first two would be caught by a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.-only gate. The third — dependent_locality going 0 → 1066 — would NOT be caught by a recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1.-only gate (recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. there actually improved, 0% → 30%). That's the v0.6.1 failure pattern from the postmortem encoded as a mechanical check: a tag silently exploding in false-positive count even as nominal recallrecallOf the spans whose gold label is a given tag, the fraction the model found. High recall means few misses. Paired with precision to compute F1. holds up.

Workflow

1. Score the baseline:

   node scripts/eval-morphology-fst.ts \
     --model $MAILWOMAN_DATA_ROOT/.../model-v060.onnx \
     --tokenizer $MAILWOMAN_DATA_ROOT/.../v0.6.0-a0/tokenizer.model \
     --model-card neural-weights-en-us/model-card.json \
     --admin-fst $MAILWOMAN_DATA_ROOT/.../fst-en-us.bin \
     --golden data/eval/golden/v0.1.2 \
     --name v0.6.0-default \
     --out-json /tmp/eval-v060.json

2. Score the candidate (same evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error., new weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.).

3. Run the gate:

   node scripts/eval-gate.ts \
     --baseline /tmp/eval-v060.json \
     --candidate /tmp/eval-v062.json \
     --out-md /tmp/gate-report.md

   Exit 0 → safe to promote. Exit 1 → block the release-it dispatch.

The gate is reusable forever — it doesn't know about specific tags or specific modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', only about the threshold contract. v0.7+ modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' with dependent_street, unit_designator, JP tags will be measured the same way.

What the gate does NOT replace

• The v0-vs-neural harness (per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. + per-file pass rates across 376 hand-tuned assertions). The gate works on aggregate golden-set metrics; the harness works on the legacy rule-based pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.'s harsh acceptance criteria. Both should be green.
• Manual demo-preset evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.. The /eval-model skill remains the human-eyeballs gate before any push to default.
• Falsehoods catalog regression check (data/eval/falsehoods/streets.jsonl).

The gate is one slice of the pre-publish protocol, not the whole thing.

See also

• Layer 1 eval — the evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.-script's --out-json flag emits the format this gate consumes
• v0-vs-neural harness — the complementary breadth metric
• 2026-05-28 night-2 postmortem — DeepSeek-turn-1 rubric this gate operationalizes