Synthetic corpus — alignment validation is essential

Both v0.5.0 corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. threadsthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. (B kryptonite and B2 transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.) used an LLM (DeepSeek) to generate annotated trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. rows. Both surfaced the same lesson: the substring-match alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. check is structural infrastructure, not a quality filter you can drop later. This article explains why.

What alignment validation does

After the LLM generates a row, the validator confirms that every annotated component literally appears in the surface string. For example, if the LLM returns:

{
	"surface": "350 5th Avenue, New York, NY 10118",
	"house_number": "350",
	"street": "5th Avenue",
	"locality": "New York",
	"region": "NY",
	"postcode": "10118"
}

the validator checks: does 350 substring-match the surface? Does 5th Avenue? Does New York? Does NY? Does 10118? If any one fails, the row is rejected before it reaches the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus..

The implementation is unglamorous (generate_deepseek_corpus.py::align_or_reject if you want to read it) but the contract it enforces is what makes the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. trustworthy.

What goes wrong without it

LLMs hallucinate around component boundaries. The B and B2 reject logs catalogue the common failure modes:

• Component embedded in another component. house_number=350, street=350 5th Avenue — the streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. contains the house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales.. A naïve trainer would see 350 as both a house numberhouse numberThe numeric or alphanumeric identifier of a building on a street. Mailwoman's house_number component; its position relative to the street name flips between locales. and as the first tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. of a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., and learn an incoherent BIO labelling.
• Hallucinated transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. fragments. For B2's Cyrillic / Japanese / Hangul output, the LLM sometimes added a transliterated suffix or prefix that does not appear in the source surface. The annotated component would substring-match against itself but not against the surface — silently misaligned data.
• Component missing from surface. The LLM produced a venue field for a string that had no venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. in it (modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' imagined one). No substring match → reject.
• Off-by-one in the annotation. house_number=35 for a surface starting 350 5th Avenue — close but wrong. Substring match fails because 35 does appear, but the validator can be made stricter to demand the full tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. (and was).

Reject ratesreject rateThe fraction of generated or harvested rows that fail validation and are excluded from the corpus — a quality gauge for a synthesis source. from the two runs:

Run	Total generated	Rejected	Rate
B (kryptonite)	4,872	101	2.1%
B2 (transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis.)	74,140	821	1.1%

Neither approaches zero. The 1–2% range is the floor for DeepSeek-as-corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. at the prompt quality we shipped with. Better prompting drops it further but does not eliminate it. The validator is permanent infrastructure.

Synthetic data needs this more than real data

For corpora harvested from real sources (NPPES, NADNAD (National Address Database). A US Department of Transportation dataset of structured address points, added to the training corpus as a major source of real US addresses., WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations., BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure.), alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. problemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. are bugs in the harvester or in the source — rare, deterministic, fixable by a one-line ETL patch. A 0.01% reject ratereject rateThe fraction of generated or harvested rows that fail validation and are excluded from the corpus — a quality gauge for a synthesis source. is plausible.

For LLM-generated corpora, alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. problemsexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed. are inherent to the generation process. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' is producing both the surface and the annotation in the same forward pass; nothing forces internal consistency. Even with temperature=0 and reasoning_effort=low, the 1–2% rate persists.

This shifts the validator from "quality filter" to "trust boundary" — the difference between trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. on data the LLM only thinks it generated correctly, and trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. on data that's provably consistent with its own annotation.

What the validator does not catch

The substring check enforces structural consistency but not semantic correctness. A row where the LLM annotated street=Main for the surface 123 Main Street will pass — Main substring-matches. But the correct annotation is street=Main Street. The check confirms what is annotated exists in the surface; it does not confirm what should be annotated matches what is annotated.

Three things bridge the gap:

1. Prompt engineering — the prompts for B and B2 included explicit examples showing full-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. annotations. This moves the LLM toward the right behaviour at generation time.
2. corpus-audit — runs over the assembled corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. and checks distributional invariants (component balance, surface length distribution, character-set coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.). It catches "annotations are technically consistent but pathological" classes of bug.
3. Held-out evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. — the golden evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set (v0.1.2) catches the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' learning bad patterns from systematically-skewed synthetic data, even when each row is individually consistent.

The validator is the first line of defence. Audit and evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. are the second and third.

Pattern to carry forward

Any future synthetic-corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. should ship with:

1. A substring-match validator that runs per-row before the row reaches the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. Reject reasons logged with structured tags so the reject-rate breakdown is greppable (reject:not-in-raw:street etc., the shape B and B2 use).
2. A reject-rate floor expectation in the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.'s README. If you see reject ratereject rateThe fraction of generated or harvested rows that fail validation and are excluded from the corpus — a quality gauge for a synthesis source. drop to zero, that is a bug in the validator, not a quality breakthrough.
3. A corpus-audit pass before declaring done — the validator is necessary but not sufficient.

The cost is low (a few hundred lines of Python); the value is high (you can trust that the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. matches its own annotations).

See also

• CORPUS_V0_4_0_GENERATION.md — the operational record for the v0.5.0 corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. generation, including the actual prompts used
• v0.5.0 — as shipped — context for where corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-v0.4.0 fits
• The knowledge ladder — why we validate generation at this layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. rather than trying to fix it downstream