WOF data pipeline — sync, prepare, resolve

Who's On FirstWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. (WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations.) is the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. underneath Mailwoman's Stage 6 resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.. It provides place IDs, parent-child chains, placetypesplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match., and multilingual name variants for every admin boundary on Earth. This article documents how the Mailwoman codebase ingests, normalises, and serves that data — layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. by layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6., from raw GeoJSON to the queryable SQLite artifacts.

If you've read Resolver and Who's On First for the runtime story, this is the build-time companion: how the data gets from raw GeoJSON on GitHub to a queryable per-placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. SQLite DB.

The end-to-end flow (as designed)

WOF GitHub repos                     Per-placetype SQLite DBs
(100K+ .geojson files)               (<placetype>/<lang>.db)
         │                                     ▲
         │ git clone / pull                    │ upsert
         ▼                                     │
┌─────────────────┐     ┌──────────────────┐   │
│ commands/wof/   │     │ commands/wof/    │   │
│   sync.tsx      │────▶│   prepare/       │───┘
│                 │     │   index.tsx       │
│ • gh repo list  │     │   _app_worker    │
│ • git clone     │     │                  │
│ • Placetype     │     │ • glob *.geojson │
│   .prepare()    │     │ • Piscina batch  │
└─────────────────┘     │ • pluckSpec      │
                        │ • write to DB    │
                        └──────────────────┘

Status (updated 2026-07-14): every stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). below shipped and matured — the unified admin gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. is now built, verified, and sealed end-to-end by the mailwoman gazetteer CLI (gazetteer build admin: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. ingest → Overture divisions → GeoNamesGeoNamesA free global gazetteer combining administrative, postal, and POI data across 200+ countries. Supplements Who's On First for postcode centroids and places where WOF has gaps. folds → enrich → FTS → verify gate → seal), which absorbed the standalone build-unified-wof and build-importance scripts described in this article's history.

Layer by layer

commands/wof/sync.tsx — repo cloning

Discovers WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. data repos via gh repo list whosonfirst-data --no-archived, optionally filtered by --repos (comma-separated allow-list). Clones or pulls each into a local directory. After all repos sync, calls Placetype.prepare() to load the placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. hierarchy from whosonfirst-placetypes.

The --repos filter exists because the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. build only needs a handful of repos (admin-us, admin-fr, postalcode-us, postalcode-fr) and cloning all ~100 WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. repos is 2.9 GB of git for no benefit.

commands/wof/mermaid.tsx — render the placetype hierarchy

Once sync has populated the local clone of whosonfirst-placetypes, wof mermaid renders the hierarchy (any subtree of it) as a MermaidMermaidA Markdown-friendly diagram syntax used in these docs for flowcharts. flowchart. Output goes to stdout by default, or to a file with --output. The --roles flag filters descendants by common, common_optional, and/or optional so you can keep charts readable when the optional tail of the tree gets noisy.

# Full tree from the root, piped to a file
node mailwoman/out/cli.js wof mermaid <localRepoDir> planet > planet.mmd

# Common-only subtree rooted at "country"
node mailwoman/out/cli.js wof mermaid <localRepoDir> country \
  --roles common,common_optional --output country.mmd

# Swap the color palette for any d3-scale-chromatic sequential interpolator
node mailwoman/out/cli.js wof mermaid <localRepoDir> planet --interpolator viridis

Edges are colored by their depth from the root so lineage paths (planet → continent → country → …) trace a smooth gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. down the chart. The default gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. is interpolateViridis (perceptually uniform, colorblind-friendly); override with --interpolator <name> to use any other d3-scale-chromatic interpolate* function (turbo, plasma, cool, magma, etc.). Node fills always use the hand-tuned blue/green/yellow role palette regardless — sampled gradientsgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. carry less semantic weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. than the hand-tuned colors for the three categorical roles, so the gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. lives on the edges only. Cyclic interpolators (rainbow, sinebow) work too but sample to similar colors at the gradientgradientThe direction and rate at which the loss would change if each parameter were nudged. Training follows the gradient downhill to reduce error. Huge gradients are tamed by gradient clipping. endpoints.

The renderer is generateMermaidMarkup in core/resources/whosonfirst/placetypes/mermaid.ts, applied to whatever Placetype.find(<name>) returns from the static cache loaded by Placetype.prepare(). The traversal is a recursive findChildren walk (the same one wof tree uses) — every emitted edge maps directly to a parent → child relationship colored by the child's role, so ELK gets a clean DAG to lay out instead of a hairball of root → every-descendant star edges. Leaf children of the root (e.g. custom, ocean, timezone under planet) sit alongside their real siblings instead of being flung to the periphery. Long classDef / linkStyle lines are written to stdout directly (bypassing Ink's <Text> renderer) so the output stays valid MermaidMermaidA Markdown-friendly diagram syntax used in these docs for flowcharts. when redirected.

commands/wof/tree.tsx — emit a nested JSON tree

Same inputs as wof mermaid (<localRepoDir> <placetype>, optional --roles, optional --output) but produces a PlacetypeTreeNode — { name, id, role, children } recursively — for consumers that expect a nested tree (e.g. d3-hierarchy). --compact switches off pretty-printing.

# Pretty tree from "country" filtered to common roles
node mailwoman/out/cli.js wof tree <localRepoDir> country --roles common

# Compact, written to file
node mailwoman/out/cli.js wof tree <localRepoDir> locality --compact --output locality-tree.json

The renderer is generatePlacetypeTree in core/resources/whosonfirst/placetypes/tree.ts — a recursive findChildren(roles) walk. Headsattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads.-up: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. placetypesplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. are a DAG with heavy descendant sharing (e.g. installation is a leaf reachable from many parents), and a tree projectiontree projectionThe process of converting a flat list of labeled spans into a nested tree representation — street contains street_prefix and street_suffix, locality contains region. Enables hierarchical address operations like containment checks. duplicates every shared subtree under every parent path. For shallow / sparse roots like country or locality the tree is tiny; for planet it explodes to ~174 MB. Use wof graph for those instead.

commands/wof/graph.tsx — emit a node-link JSON graph

The flat sibling to wof tree. Emits { root, nodes: [...], links: [{ source, target }] } — each node and each parent→child edge appears exactly once, regardless of DAG topology. planet is 9.4 KB in this shape vs 174 MB nested. Field names follow d3-force / react-flow / cytoscape conventions so the output drops straight into common HTML graph viewers.

# Full planet graph, compact JSON, ~9 KB
node mailwoman/out/cli.js wof graph <localRepoDir> planet --compact --output planet-graph.json

# Common-only subgraph rooted at "country"
node mailwoman/out/cli.js wof graph <localRepoDir> country --roles common

The renderer is generatePlacetypeGraph in core/resources/whosonfirst/placetypes/graph.ts. Same recursive findChildren(roles) walk, but tracks visited nodes and emitted (parent, child) pairs in sets so duplication is impossible. Stays O(nodes + edges) for any root.

commands/wof/prepare/index.tsx — batch GeoJSON processing

The heavy-lift command. Designed to process 100K+ individual .geojson files (one per WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. featurefeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them.) using Piscina worker threadsthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases..

Architecture:

• Main threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. globs **/*.geojson from the admin directory via fast-glob streaming.
• Files are batched via takeAsync(matchStream, BATCH_SIZE) — yields arrays of filenames, one per available CPU core.
• Each batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. is dispatched to workers via takeInParallel(fileNames, BATCH_SIZE, delegateInsertion).
• Workers run _app_worker.mts::insertRecord(filePath): reads the file, parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. GeoJSON, calls pluckPlacetypeSpec to extract structured fields.

What pluckPlacetypeSpec produces:

interface ParsedWOFPlacetype {
	id: number
	parent_id: number
	name: string
	src: string
	placetype: WhosOnFirstPlacetype
	localizedPropMap: Map<Alpha3bLanguageCode, Map<WOFNameKind, string>>
}

The localizedPropMap is the multilingual name-variant surface — for each language code (ISO 639-3b), it carries preferred, variant, colloquial, abbr, and short forms. This is exactly what PlacetypeDataSource.records expects in its columns.

The worker writes to PlacetypeDataSource (per-placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. SQLite mini-DBs). An earlier iteration targeted Redis; that was replaced in v0.5.0.

The incomplete "send batches of filenames" design

The original intent was more efficient than the current per-file dispatch:

1. Main threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. globs a batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. of N filenames (N = availableParallelism()).
2. Sends the entire batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. to ONE worker (not one filename per Piscina task dispatch).
3. Worker processes the batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. sequentially (file reads are fast once in cache), builds up an in-memory table of results.
4. Worker returns the batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. results to the main threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. (or writes directly to an in-memory SQLite, then flushes).

This design reduces IPC round-trips (one per batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. vs one per file) and lets the worker keep a warm DB connection open across the batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. instead of opening/closing per row. The per-file dispatch shape currently in the code was the first iteration; the batched-filename shape was the intended next step.

The in-memory-SQLite-then-consolidate idea

A related design that never took shape: each Piscina worker holds its own :memory: SQLite database during the batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU., inserts all processed records into it (zero disk I/O during the hot path), then at batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.-end dumps the in-memory DB to a temporary file and signals the main threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases.. The main threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. then merges N temporary DB files into the final per-placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match./per-language DBs on disk via ATTACH DATABASE + INSERT INTO ... SELECT FROM ....

Why in-memory first:

• Avoids disk thrashing during the hot path (100K individual writes to the same few DB files from N concurrent workers would serialize on SQLite's WAL writer).
• Workers are completely independent — no shared file handles, no lock contention during processing.
• Consolidation is a single bulk operation after all workers finish, which SQLite handles efficiently via ATTACH.

This never went past the design stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone).. The Redis target worked well enough for prototyping, and by the time it was clear Redis wouldn't survive into production, other priorities (the neural classifierneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package., resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. integration) took precedence.

PlacetypeDataSource — the target schema

Per-placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match., per-language SQLite DB files at <dataDir>/<placetype>/<lang>.db. Schema:

CREATE TABLE records (
  'id'        INTEGER NOT NULL,
  'src'       TEXT NOT NULL,
  'name'      TEXT NOT NULL,
  'preferred' TEXT,
  'variant'   TEXT,
  'colloquial' TEXT,
  'abbr'      TEXT,
  'short'     TEXT,
  'parent_id' INTEGER,
  PRIMARY KEY ('id', 'src', 'name')
);

• id: WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. Brooklyn Integers ID.
• parent_id: the hierarchy chain for concordanceconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement. scoring.
• name + variant columns: the alias-resolution surface the reconciler needs ("Saint Petersburg" vs "St. Petersburg" vs "St Petersburg" are name, variant, and short for the same ID).
• src: provenance tracking (whosonfirst, quattroshapes, etc.).

DataSourceCache manages a pool of open PlacetypeDataSource handles, keyed by (placetype, languageCode). Lazy-opens on first access; disposable.

Placetype — the hierarchy cache

In-memory hierarchy built from the whosonfirst-placetypes codex (a JSON spec defining the parent/child/sibling graph of all WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. placetypesplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match.). Placetype.prepare() reads the JSON files from the cloned repo and populates static maps (#byID, #byName, #childrenOfParentName).

This is the placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match.-level hierarchy (countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. → regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. → county → localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. → neighbourhood), not the per-record parent_id chain. The per-record parent_id chain lives in PlacetypeDataSource.

To eyeball this hierarchy, use wof mermaid for a flowchart view, wof graph for a flat node-link JSON, or wof tree for a nested form (small subtrees only). All three pull from the same in-memory cache.

WOFPlacenameCache + loader.ts — the GeoJSON-direct index

A separate path that builds a normalised placename→languages index directly from the GeoJSON source files via fast-glob + TextSpliterator. Used by the rule-based classifiersrule-based classifierMailwoman's legacy v0 parser — a library of deterministic token classifiers (house number, street suffix, postcode, place name, etc.) composed by priority. Now primarily used for corpus labeling, fallback classification, and arbitration diagnostics. (localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy./regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. dictionaries) to answer "is this string a known placename in any language?" without going through SQLite.

This is the path that works today. It reads raw GeoJSON and builds Map-based indexes in memory. It doesn't use PlacetypeDataSource. The two paths (WOFPlacenameCache for rule classifiers, PlacetypeDataSource for the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon./reconciler) were always intended to coexist — different query patterns, different storage trade-offs.

Where Spliterator fits

The spliterator package (published separately at @sister-software/spliterator) was born from this pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package.'s needs:

• TextSpliterator — line-delimited iteration over file streams without loading entire files into memory. Used by loader.ts for reading the large WOF GeoJSONWOF GeoJSONThe raw one-feature-per-file GeoJSON distributed by Who's On First repositories — the input that Mailwoman's WOF SQLite build consumes. bundles.
• AsyncSpliterator.asMany(source, delimiter, concurrency) — splits a single large file into N concurrent byte-range iterators for parallel processing. Designed for the case where WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. data arrives as a single NDJSON dump (e.g. from data.geocode.earth) rather than 100K individual files. Marked @internal; never exercised at scale because the individual-file path was sufficient for the repos we actually use.
• asyncParallelIterator / takeAsync — the batching primitives that commands/wof/prepare uses to dispatch work to Piscina workers. takeInParallel in core/resources/collections.ts is the local equivalent (bounded-concurrency async generator).

The relationship: Spliterator handles the read parallelism (splitting a file into chunksunknown tokenA placeholder for input not in the tokenizer's vocabulary. Byte fallback largely removes the need for one by encoding unseen characters as raw bytes instead.); Piscina handles the compute parallelism (processing parsed records across CPU cores); PlacetypeDataSource handles the write (structured storage). The pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. needs all three layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6.; today only the first two are wired.

How this connects to the reconciler

The Stage 5 joint decoderdecoderIn a transformer encoder-decoder model, the part that produces output sequences. Mailwoman's classifier is encoder-only (no decoder); the 'CRF decoder' is a different thing — a structured-prediction layer that picks the best label sequence from the encoder's outputs. (reconcileSpans in core/pipeline/reconcile.ts) needs three inputs per spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree.:

1. Phrase proposals — from the phrase grouperphrase grouperStage 2.7 of the runtime pipeline: proposes coherent input units (street phrase, locality phrase, postcode, etc.) with structural kind hypotheses. Decouples boundary discovery from type classification so the classifier answers 'what type?' not 'where?' (shipped, working).
2. Classifier top-k — from per-spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. logitlogitA raw, unnormalized per-label score the model outputs before softmax. Priors and biases are added in logit space, then softmax turns logits into probabilities. aggregation (being wired now, PR in progress).
3. ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. candidates + parent chainsparent chainThe sequence of administrative parents above a place — Springfield → Sangamon County → Illinois → United States. Used to check the geographic coherence of a parse. — from PlacetypeDataSource.

Item 3 is what this pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. produces. Specifically:

• Name lookup: given a spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. body like "Houston", find all PlacetypeRecord rows where name = 'Houston' OR variant = 'Houston' OR abbr = 'Houston' etc. Filter by placetype (e.g. only locality candidates for a spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. the classifier tagged as localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.).
• Disambiguation: when multiple candidates exist (there are 13 "Houston" entries in WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. US admin), filter by parent_id — the classifier's regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. tag for the same input narrows the candidate set.
• Parent-chain walk: for concordanceconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement. scoring, walk parent_id up the hierarchy until reaching country. The chain Houston (id=3) → Texas (id=2) → United States (id=0) is what tells the reconciler that locality=Houston, region=TX is coherent.

The WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. spot-check (2026-05-24) confirmed the raw spr table's parent_id chains are trustworthy (18/20 correct) but surfaced two findings the prepare pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. needs to address:

1. Name aliasing: "Saint Petersburg" not found via bare name lookup — a known WOF gotcha — WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. stores it as "St. Petersburg". The variant/short columns in PlacetypeDataSource are designed for exactly this; they just need to be populated.
2. Disambiguation: bare-name lookup without parent_id filtering picks the wrong record (a Nebraska village named "New York" outranks NYC by alphabetical ID ordering). The find query needs a parent_id filter or population-weighted ranking.

Both findings are closed by completing the prepare pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. (populates the variant columns) and using PlacetypeDataSource.find with appropriate criteria (filters by parent_id or placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match.).

What needs to happen next

In priority order:

1. Migrate the worker target from Redis to SQLite. Recommended shape: worker opens PlacetypeDataSource per (placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match., language) on demand, batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.-inserts via a wrapped transaction (1000 rows per COMMIT for throughput). Workers operate on separate DB files so no cross-process locking.

2. Batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. filenames to workers instead of dispatching one-at-a-time. Reduces IPC overhead. Each Piscina task gets an array of file paths; worker processes them sequentially with a warm DB connection.

3. Wire PlacetypeDataSource into the reconciler as the ResolverCandidatesLookup + ParentChainLookup implementation. This replaces the mocked resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. in the reconciler's current test surface with real data.

4. (Stretch) In-memory-then-consolidate pattern for maximum throughput. Workers write to :memory: during the hot path, dump to temp files at batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.-end, main threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. merges via ATTACH. Only worth pursuing if step 1+2 are too slow for the full WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. dataset (~120K featuresfeatureAn input signal a model conditions on. Beyond the raw tokens, Mailwoman feeds soft features — gazetteer-membership channels and the postcode anchor — that inform predictions without overriding them. × 5-10 languages = 600K-1.2M rows).

See also

• Resolver and Who's On First — the runtime resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. that consumes this data
• The knowledge ladder — StagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 6 in the pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. decomposition
• Joint decoding — a walkthrough — how the reconciler uses resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. candidates + parent chainsparent chainThe sequence of administrative parents above a place — Springfield → Sangamon County → Illinois → United States. Used to check the geographic coherence of a parse.
• STAGES.md — formal per-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). type contracts