Skip to main content

Keep your data current

Outcome. You can tell which vintage of each data artifact is on disk, refresh one that has changed, and switch a long-running process to a new version without restarting it.

One thing to know before the steps: mailwoman data pull checks whether a file is there, not whether it is current. mailwoman data status is the freshness check. Running data pull on a stale file reports it as already present and downloads nothing.

Prerequisites​

  • A data root, and $MAILWOMAN_DATA_ROOT pointing at it. Every transcript below uses /tmp/mw-fresh-demo.
  • At least one bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. already pulled. Your first ten minutes pulls candidate; Improve geocode precision pulls the per-state shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row..
  • mailwoman 8.7.0 or later. data pull and data status do not exist before it.

1. Read what you have​

mailwoman data status candidate
✓ candidate: wof/candidate.db: present (1652.9 MB) at /tmp/mw-fresh-demo/wof/candidate.db
PASS (1/1 checks)

With no bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. named, data status reports on all four (candidate, poi, fr, us) and prints one line per artifact — 106 lines, because the us bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. is 103 files. Name the bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. you care about.

The check is a byte-count comparison against the size recorded for the published artifact, so it catches a truncated or half-copied file:

mailwoman data status candidate
✗ candidate: wof/candidate.db: stale — 900.0 MB on disk vs 1652.9 MB (recorded) at
/tmp/mw-fresh-demo/wof/candidate.db
FAIL (0/1 checks)

The exit code is 0 only when every artifact is present and matches. That makes it usable as a deploy gate.

2. Refresh what has moved​

A stale artifact is the case data pull will not fix on its own:

mailwoman data pull candidate --dry-run
✓ candidate: gazetteer/2026-07-07a/candidate.db: already present (/tmp/mw-fresh-demo/wof/candidate.db) —
skipped
PASS (1/1 checks)

That transcript was produced against the 900 MB truncated file from step 1. The skip is a filesystem existsSync, nothing more. --force is what makes the download happen:

mailwoman data pull candidate --dry-run --force
✓ candidate: gazetteer/2026-07-07a/candidate.db: [dry-run] 1652.9 MB
https://public.sister.software/mailwoman/gazetteer/2026-07-07a/candidate.db →
/tmp/mw-fresh-demo/wof/candidate.db

Drop --dry-run to run it. The download streams to <data-root>/tmp/, is verified against the Content-Length the server sent, is sealed, and only then moves into place — so a failure partway through leaves the old file untouched rather than a corrupt new one. --dry-run makes no network call at all: the plan comes from the registry and a filesystem stat.

For the us bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line., --only scopes the refresh to one state rather than all 103 files:

mailwoman data pull us --only dc --force

3. Ask the bucket instead of the registry​

The size data status compares against is the one recorded when the bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. was surveyed. --check-remote replaces it with a live HEAD request per artifact:

mailwoman data status candidate --check-remote
[DEBUG] (mailwoman data status): HEAD: https://public.sister.software/mailwoman/gazetteer/2026-07-07a/candidate.db
[DEBUG] (mailwoman data status): 200 (uncached) HEAD: https://public.sister.software/mailwoman/gazetteer/2026-07-07a/candidate.db
✓ candidate: wof/candidate.db: present (1652.9 MB) at /tmp/mw-fresh-demo/wof/candidate.db
PASS (1/1 checks)

The visible difference is one word in the stale line — (live) where the offline run says (recorded). Only artifacts already on disk are probed, so the flag stays cheap even against the us bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line.: a missing file is reported from the registry without a request.

Two limits on what this proves. No bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. publishes an md5 sidecar today, so the comparison is byte count and nothing stronger; a rebuild that landed at the same size would pass. And the remote path is dated and immutable (gazetteer/2026-07-07a/candidate.db), so a live HEAD confirms your copy of that artifact, not that a newer artifact exists. A new build is a new path, which arrives in a new release of the mailwoman package.

4. There is no refresh cadence​

No cadence is committed. The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. and layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. artifacts are rebuilt when coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. changes or an ingest bug is fixed, not on a schedule, and no schedule is promised. This is the same statement the pricing page makes, and it is the one to plan against: do not build a pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. that assumes a weekly or monthly artifact.

What you get instead is that every artifact is sealed and dated. The vintage travels in the remote path, so mailwoman data pull candidate --dry-run always tells you which build your install is pinned to without downloading anything.

5. Pin a version, and switch without a restart​

The per-state shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. support a second addressing scheme. Instead of one file per state, publish <family>/<family>-us-<slug>-<version>.db alongside the old one and put a releases.json at the data root naming the current version per family:

{ "address-points": "2026-05-20.0", "interpolation": "TIGER2023" }

ShardProvider reads that manifest when it is constructed:

import { AddressPointSqliteLookup, StreetInterpolator } from "@mailwoman/resolver-wof-sqlite"
import { ShardProvider } from "mailwoman/geocode-core"

const shards = new ShardProvider({ AddressPointSqliteLookup, StreetInterpolator }, "/tmp/mw-releases-demo")

console.log("versions:", JSON.stringify(shards.versions()))
console.log("dc address points open:", Boolean(shards.for("dc").addressPoints))

With no manifest, it runs in legacy mode against the unversioned filenames:

versions: null
dc address points open: true

With a manifest naming a version whose file exists, it opens that file:

versions: {"address-points":"2026-05-20.0"}
dc address points open: true

A pinned version whose file is absent falls back to the unversioned name and says nothing. That is the trap worth knowing. Above, a run against a data root holding both address-points-us-dc.db and address-points-us-dc-2026-05-20.0.db, pinned to a third version that was never published:

versions: {"address-points":"2026-06-01.0"}
dc address points open: true

The shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. opened. It is the legacy file, not the pinned one. Remove the legacy file and the same pin resolves to nothing at all:

versions: {"address-points":"2026-06-01.0"}
dc address points open: false

So a typo in releases.json degrades to the old data rather than to an error. Assert on versions() after deploy rather than assuming the pin took.

The reason to run the manifest at all is reload(). Write the new shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. files alongside the old, then write releases.json — one atomic file write — then call shards.reload(). Any shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. whose resolved path changed is reopened; any shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. whose path is unchanged keeps its open handle. Retired handles close on the next reload, which gives in-flight reads a generation of grace.

// A long-running server, on a signal or a poll.
process.on("SIGHUP", () => {
	console.log("switched to", JSON.stringify(shards.reload()))
})

Verify​

A deploy check that fails when any artifact you depend on has drifted:

mailwoman data status candidate --check-remote && echo "data OK"
✓ candidate: wof/candidate.db: present (1652.9 MB) at /tmp/mw-fresh-demo/wof/candidate.db
PASS (1/1 checks)
data OK

Limits​

  • data status compares against the published artifact, not against validity. A shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. you built yourself from the US dataset tutorial fails the check by construction, because its byte count is not the published one. That page explains the case.
  • The four bundlesevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. are the whole published set. There is no timezone, NUTS or UN/LOCODE artifact in the bucket; those packages build their data locally. mailwoman data pull with any other name exits 1 and lists the four.
  • releases.json addresses the per-state families only. The versioned scheme covers address-points and interpolation. candidate.db, poi.db and the French shardshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. are addressed by their dated remote path instead, so refreshing one of those means upgrading mailwoman to a release that registersinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. a newer path, then pulling again.
  • --check-remote needs network egress to public.sister.software. In an air-gapped deploy the offline comparison is what you have, and it still catches truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap..
  • Improve geocode precision — which bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line. buys which resolution tier.
  • Build the US address dataset from public sources — what to do instead when you build shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row. rather than pull them.
  • Deploy on a serverless runtime — how the same artifacts behave on a read-only, short-lived filesystem.