14 posts tagged with "Geocoding"

Last night the plan was straightforward: mailwoman was missing a pile of non-US coverage, and we were going to fix the parser to close the gap. By morning we'd shipped something better than planned, killed the original idea, and caught ourselves about to spend a GPU budget on a bug that lived in a symlink. Here's how a night of "fix the model" turned into "stop trusting your evals."

You can point a Nominatim client at Mailwoman now. Same endpoints — /search, /reverse, /status — same JSON shape, so your existing code doesn't change. The difference is what's behind it: no PostgreSQL, no osm2pgsql import that takes hours and tens of gigabytes, no server you can't ship inside an app. A geocoder that answers Nominatim's questions from a SQLite file.

You've got a hundred thousand addresses to reconcile. Two databases, the same clinics and providers scattered across both, each one spelled a dozen ways: abbreviated here, reordered there, a postcode dropped, a suite number glued to the street. You run them through a geocoder, match on the resolved coordinate, and it works. Mostly. Some fraction land on the wrong building, the wrong block, the wrong town, and the geocoder won't tell you which fraction. It hands back a pin for every row and the same silent confidence for all of them: none.

That's the gap we set out to close. Every geocoder chases accuracy, be-right-more-often, and so do we. The piece almost nobody hands you is the one underneath it: a number on each answer that tells you which ones to trust, so you can keep the good ones and send the rest to a human.

Open the demo, type an address in Kabul, and watch nothing happen. Not a wrong pin a few streets off. Not a city-center fallback. Nothing — the gazetteer has never heard of the place. Try Hong Kong. Try Tirana, or Chişinău, or anywhere in the Democratic Republic of the Congo. Same silence. We had been calling this gazetteer "world coverage" for weeks, and it covered 97 of the world's ~195 countries. The other ninety-odd were simply not in the file.

The whole geocoder runs in your browser. You type an address, you get a rooftop coordinate, and no server ever sees your query — the gazetteer it resolves against is a SQLite database sitting on a CDN, and the page reads it with HTTP range requests, a few kilobytes at a time. It's a lovely trick. We were proud of it. Then we counted the requests it took to find a single city, and the number was 243.

So the questions for the day: why does looking up one name cost 243 round trips? What goes wrong when you search a database you can only read a slice at a time? And how do you get a global gazetteer — every country, region, county, and city we resolve against — down to about a dozen reads without putting a server back in the loop?

Overnight I taught the geocoder Canada. Or I thought I did. I pulled 843,000 Canadian postal codes, computed a centroid for every one, spot-checked the result — M5H 2N2, downtown Toronto, 43.652, −79.382, dead on — and validated the database every way I could read it: every postcode present, every coordinate right, nothing else disturbed. Green, top to bottom. This morning, before flipping it live, I asked the demo to find a Toronto address. It dropped the pin in Ohio.

So, the questions for the morning. How does a database that passes every check still land the answer 600 kilometres wrong? What does it actually mean to "validate" a thing? And why is Toronto in Ohio?

Open the demo, open your browser's network tab, and type an address. You'll watch it resolve to a rooftop coordinate — 1600 Pennsylvania Ave lands on the actual building, within about ten metres — and then you'll notice what's missing from the network tab: a request carrying your address. There isn't one. The parser ran in the page. The gazetteer it resolved against is a file on a CDN that the page read a few kilobytes at a time. The query never left your machine.

That's the pitch, and it's worth being clear about why it's unusual, because the three things the rest of the market hands you each ask you to give something up.

Every geocoder turns an address into a coordinate. Almost none of them will tell you where that coordinate came from. You get a latitude, a longitude, and a vague confidence enum, and when it's wrong you have no thread to pull — no way to know whether the point came from a federal data release, a county GIS office, or a straight line drawn down the middle of a street. Mailwoman keeps the source on every point. Here's New York, every dot colored by the open dataset it came from.

Three public datasets land on your desk. The national provider registry — NPPES, every NPI in the country. A federal telecom-funding file from the FCC's Rural Health Care program, one slice of the Universal Service Fund. A state list of licensed nursing facilities from Texas HHSC. You want to know which records describe the same provider, and not one of the three shares an identifier with the other two. The NPI is internal to NPPES. The funding file keys on its own SPIN. The state list has its own facility ID. There's no crosswalk, because nobody ever built one.

So you do what everyone does: you start a spreadsheet, you sort by name, and you give up around row 400.

You have a pile of records and no key to join them on. A clinic shows up in the federal provider registry, again in a state licensing export, a third time in a funding-program spreadsheet somebody keyed by hand. None of those files share an identifier. The provider number is internal to one publisher, the facility ID to another. So the join you actually want — which of these are the same place — isn't a join at all. It's a judgment call, repeated a few million times.

The market hands you two tools for this, and each one solves a different half of the problem. Neither one finishes.