Skip to main content

Gazetteer-first geocoding

The most radical simplification: don't parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. at all. Treat the input as an information retrieval problem. Tokenize the string, try every tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. and n-gramn-gramA contiguous run of n tokens (a bigram is 2, a trigram is 3). Counting n-grams is a simple, pre-neural way to model which token sequences are common. against a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. index, return the best placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. match. The parser is a search engine. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. is the search result.

The approach​

This is the architecture behind Airmail, the Rust tantivy-based geocoder that sidesteps parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. entirely.

  1. Tokenize the input. Split on whitespace and punctuation. Produce every n-gramn-gramA contiguous run of n tokens (a bigram is 2, a trigram is 3). Counting n-grams is a simple, pre-neural way to model which token sequences are common. up to some reasonable limit (trigrams are usually sufficient for place namestoponymA proper name for a geographic place.).
  2. Query a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. index. The index contains every place nametoponymA proper name for a geographic place. from Who's On FirstWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations., OpenStreetMapOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names., GeoNamesGeoNamesA free global gazetteer combining administrative, postal, and POI data across 200+ countries. Supplements Who's On First for postcode centroids and places where WOF has gaps., or another gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.. Each entry has a name, placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match., parent chainparent chainThe sequence of administrative parents above a place — Springfield → Sangamon County → Illinois → United States. Used to check the geographic coherence of a parse., and coordinate. The index supports prefix search and fuzzy matching.
  3. Rank results. Score each hit by: name match quality (exact > prefix > fuzzy), placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. priority (localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. > regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. > countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., or configurable), population (larger places score higher, all else equal), and tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. (an n-gramn-gramA contiguous run of n tokens (a bigram is 2, a trigram is 3). Counting n-grams is a simple, pre-neural way to model which token sequences are common. that covers more of the input scores higher).
  4. Return the best placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. match. The result is a place with a placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match., coordinate, and parent chainparent chainThe sequence of administrative parents above a place — Springfield → Sangamon County → Illinois → United States. Used to check the geographic coherence of a parse.. The system does not distinguish between "this tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. is a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels." and "this tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. is a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy." — it only knows "this n-gramn-gramA contiguous run of n tokens (a bigram is 2, a trigram is 3). Counting n-grams is a simple, pre-neural way to model which token sequences are common. matches a place in the index."

It avoids every parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. problem by not parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.: the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. knows what places exist, the search engine finds which place namestoponymA proper name for a geographic place. appear in the input, and the ranker picks the best one.

When it works​

  • You only need administrative places. CountrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., neighbourhood — these are in the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.. StreetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., building numbers, and venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. names are not (or are in a different index). If your use case is "which citylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy. is this in?", gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-first works.
  • You need global coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.. The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. covers every countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head.. The search engine handles every script (the index stores UTF-8). No per-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. regexes, no per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. rules, no language-specific heuristics. Add a new countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. by adding its gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. data.
  • You have ambiguous input. Springfield returns 34 candidates, ranked by population. The user or downstream system picks. The search engine doesn't need to resolve ambiguity — it surfaces it.
  • You want to avoid the parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. problem entirely. No tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-type classification, no BIO labelsBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components., no CRFCRF (Conditional Random Field). A statistical modeling method that predicts structured outputs by modeling dependencies between adjacent labels. Mailwoman uses a linear-chain CRF as the Viterbi decoder at inference time to enforce BIO label consistency — a B-street must be followed by I-street or O, never I-locality., no solver, no policy registrypolicy registryThe per-component table that decides which classifier (rule or neural) has authority for each address component. The Ship-of-Theseus dial.. Just a search index and a ranking function. The architecture is a few hundred lines of code plus a pre-built index.
  • Your addresses are short and place-name-heavy. Paris, France — two tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., both in the index. West 26th Street, New York, NY 10010 — many tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words., some in the index (New York, NY), some not (West, 26th, StreetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels., 10010). The more of the input that is place namestoponymA proper name for a geographic place., the better this works.

What you lose​

  • StreetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. and building numbers. 123 Main St, Springfield, IL — the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. finds "Springfield, IL." "123 Main St" is not a place nametoponymA proper name for a geographic place., so it's ignored. The geocoder returns the city centroidlocality centroidThe representative centre point of a city or locality, used as a coarse coordinate when no exact address point is available — the coarsest tier of the geocode cascade.. If you need streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.-level or building-level accuracy, this architecture does not provide it.
  • VenuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label. names. Empire State Building, NYC — the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. finds "NYC." "Empire State Building" is not a place nametoponymA proper name for a geographic place. in most gazetteersgazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. (WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. has some landmarks, but coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. is sparse). The geocoder returns the city centroidlocality centroidThe representative centre point of a city or locality, used as a coarse coordinate when no exact address point is available — the coarsest tier of the geocode cascade., not the building.
  • Component-level output. The result is "Springfield, IL" — a place, not a parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.. The system does not tell you which tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. were the localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy., which were the regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality., and which were noise. If you need structured output ({locality: Springfield, region: IL, street: Main St}), you need a parser.
  • TokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. as a signal. Paris, France and Paris, Texas both match a localitylocalityThe city / town / settlement component of an address: a populated place sitting between region and neighbourhood in the hierarchy.. The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. returns both. The ranker picks Paris, France (higher population). If the input also contains Texas, the ranker should boost Paris, Texas — but only if the ranker knows that Texas is a regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. that contains Paris, Texas. Gazetteer-first architectures can add this as a post-ranking filter, but it's not part of the core approach.
  • Addresses that are not place namestoponymA proper name for a geographic place.. 50 miles West of Socorro, New Mexico — the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. finds "Socorro, New Mexico." "50 miles West of" is not a place nametoponymA proper name for a geographic place. and is dropped. The geocoder returns Socorro, which is 50 miles wrong. Directional descriptions are invisible to the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture..

Where Mailwoman fits​

Mailwoman and gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-first architectures are investigating the same problem from opposite directions. GazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-first says "skip parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates., search the index." Mailwoman says "parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. first, then search the index with structured queries." Both converge on the same resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. — a WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. SQLite (or tantivy) index queried with place namestoponymA proper name for a geographic place..

Mailwoman's parser produces structured components (locality=Springfield, region=IL) that the resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. uses for constrained lookup. The gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-first approach produces unstructured n-gramn-gramA contiguous run of n tokens (a bigram is 2, a trigram is 3). Counting n-grams is a simple, pre-neural way to model which token sequences are common. matches that the ranker evaluates. The structured approach is more precise when the parser is correct; the unstructured approach is more robust when the parser is wrong.

A hybrid is possible: use gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-first as a fallback when the parser's confidence is low. If the parser returns locality=??? confidence=0.3, fall back to gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture.-first on the raw input and return the best placetypeplacetypeThe Who's On First hierarchical classification of places: planet → continent → country → region → county → locality → neighbourhood. The resolver uses placetype to rank candidates — an exact locality match outranks a county-level match. match. The parser handles the well-formed addresses; the gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. handles the messy ones.

See also​