Importance vs Population

Two signals in the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon./FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. distinguish "the famous one" from "the tiny same-name peer." They look similar but they're built from different sources and used at different layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6..

Quick reference

Signal	Table	UnitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise.	Source	CoveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.	Where used
Population	place_population	raw count of people	WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. wof:population or gn:population GeoJSON property	~15% of localities	ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. populationBoost; FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. builder fallback
Importance	place_importance	float [0, 1]	Nominatim's wikimedia-importance.csv.gz joined via WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. concordancesconcordanceA joint-decode signal that rewards a parse whose spans form a consistent Who's On First parent-child chain and vetoes contradictory ones — a hard veto for conflicts, a log-space bonus for full agreement.	Places with Wikipedia articles (varies by countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.)	FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. builder primary signal

Build pipeline

place_population is built during the main WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. SQLite build (scripts/build-unified-wof.ts). When the script reads each WOF GeoJSONWOF GeoJSONThe raw one-feature-per-file GeoJSON distributed by Who's On First repositories — the input that Mailwoman's WOF SQLite build consumes. file, it pulls wof:population (preferred) or gn:population (fallback) from the properties block and stores it in place_population.population. No external download needed.

place_importance is built by a separate script (scripts/build-importance.ts) after the main DB exists — see Wikipedia importance scores for the full ETL this feeds. It:

1. Downloads wikimedia-importance.csv.gz from nominatim.org/data/.
2. Parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. the TSV: (language, title, importance_score, wikidata_id).
3. Joins to WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. via concordances table (other_source = 'wd:id').
4. Writes one row per (WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. place, importance) match.

The Wikipedia importanceWikipedia importanceA place-notability score derived from the count and language spread of a place's Wikipedia articles, normalized to [0, 1]. A resolver ranking prior that helps pick the prominent same-named place. score is derived from Nominatim's algorithm — Wikipedia article rank, language-spread bonuses, and curation flags. NYC scores ~1.0; an unincorporated hamlet scores 0 (no Wikipedia article).

How they relate

When the FST builder loads importance scoresimportance scoreA precomputed per-place prominence score (blending Wikipedia importance with population) used to rank same-name gazetteer candidates., it follows this fallback chain (resolver-wof-sqlite/fst-builder.ts:134-152):

1. If place_importance exists: use it directly. Each WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. place gets an importance ∈ [0, 1] from Wikipedia.
2. Else if place_population exists: synthesize:

   importance = min(1.0, log2(1 + population/1000) / 14)

   A 1M-population citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. gets ~0.71, a 10K-population town gets ~0.24, missing population gets 0.
3. Else: importance = 0 for all places. FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. still works, but ranking has no famousness signal.

This means the FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. always ships a single importance field per place. The values are scaled but not directly comparable across the two paths — Wikipedia importanceWikipedia importanceA place-notability score derived from the count and language spread of a place's Wikipedia articles, normalized to [0, 1]. A resolver ranking prior that helps pick the prominent same-named place. is a curated score, log-population is a structural proxy.

Where each is used

Population

• ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. populationBoost (resolver-wof-sqlite/lookup.ts:121): adds populationBoost * min(1, log10(1+pop)/6) to the FTS5FTS5SQLite's built-in full-text-search module (with BM25 ranking). Mailwoman uses it for prefix and name matching against the gazetteer. candidate score. Default populationBoost = 4.0. Intentionally large to compensate for BM25's bias against famous places (their alt_names column inflates document length, hurting their BM25 score).
• FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. builder fallback (above).

Importance

• FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. place entry (resolver-wof-sqlite/fst-types.ts): every accepted FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. match carries an importance field. Consumers (autocomplete, query-shape ranking) sort by this.
• fst-autocomplete.ts: ranks suggestions importance-descending.
• Neural modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' emission prioremission priorA log-probability bias injected into the model's emission logits from an external signal — gazetteer frequency, an FST, a Wikipedia-importance score — combined with the learned weights at decode time. (neural/fst-prior.ts): FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. hits become BIO labelcomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. biases; importance scales the bias magnitude.

Why both exist

The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. doesn't read place_importance directly — it operates one layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. below the FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. and uses population as its tiebreaker. This means:

• For FSTFST (finite-state transducer). A compact automaton that reads an input sequence and emits an output sequence. Mailwoman encodes gazetteer names and street affixes as FSTs for fast prefix matching and prior injection without search overhead. queries (autocomplete, prefix matching, emission priorsemission priorA log-probability bias injected into the model's emission logits from an external signal — gazetteer frequency, an FST, a Wikipedia-importance score — combined with the learned weights at decode time.): importance dominates.
• For resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. queries (the post-classification step that picks "the actual place"): population dominates.

This is structural duplication, but the two signals are not redundant — they fail differently:

• A small but Wikipedia-notable village can have importance > 0 and population = NULL.
• A large agricultural localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. can have population > 100K and importance = 0 (no English Wikipedia article).

Known issues

Resolver's BM25 length penalty

The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon.'s populationBoost: 4.0 exists because SQLite FTS5FTS5SQLite's built-in full-text-search module (with BM25 ranking). Mailwoman uses it for prefix and name matching against the gazetteer.'s BM25 normalizesnormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input. by document length. The place_search table concatenates ALL multilingual name variants into a single alt_names column. Famous places have huge alt_names (NYC's is ~10K characters with 100+ languages); same-name peers have tiny alt_names. BM25 prefers the document with proportionally-more matches per tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., which can rank a 50K-population namesake above the 8.8M-population citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. the query almost certainly means.

Attempts at column-weighted BM25 (bm25(place_search, 1, 10, 1) — name 10×, alt_names 1×) don't fix this: FTS5FTS5SQLite's built-in full-text-search module (with BM25 ranking). Mailwoman uses it for prefix and name matching against the gazetteer.'s doc-length normalization uses the row's total content length across all columns, not just the weighted ones. A row with bloated alt_names is penalized everywhere, even when scoring the name column in isolation.

The structural fix is to split alt_names into a separate FTS5FTS5SQLite's built-in full-text-search module (with BM25 ranking). Mailwoman uses it for prefix and name matching against the gazetteer. table with its own length statistics. Until then, populationBoost: 4.0 is a tuning value that works for most cases but fails for the worst extremesexpectation-maximizationAn iterative algorithm that estimates model parameters when some variables are unobserved. In Mailwoman's matcher, EM learns the Fellegi-Sunter m and u parameters from unlabeled data — no training labels needed.; NYC vs West New York is the standing example.

Importance for non-English places

Nominatim's importance index is multilingual but biased toward languages with strong Wikipedia presence. Japanese localities have lower importance scoresimportance scoreA precomputed per-place prominence score (blending Wikipedia importance with population) used to rank same-name gazetteer candidates. than population-equivalent US/EU peers because the JP Wikipedia is smaller. When v0.6.0 ships JP support, importance may need a JP-specific scaling pass.