12 posts tagged with "Advanced"

Feed an address into a sequence labeler and part of it goes missing. The model tags the tokens it recognizes — house number, street, city — and the characters it doesn't understand fall on the floor. decodeAsJSON hands you back a tidy object with every field in its place, and nothing in that object tells you the model just shrugged at a fifth of the input.

So you ship that parser. It looks great on the addresses you tested it against. And you have no idea what it's dropping in production, because the output format was built to hide exactly that.

Type Vienna, Austria into our geocoder and, until this week, it answered with a confident set of coordinates: 39.32, −81.54. That's Vienna, West Virginia — population about ten thousand, a few miles up the Ohio River from Parkersburg. The capital of Austria is 7,500 kilometers and one ocean away, and the geocoder had no doubt whatsoever.

Last night the plan was straightforward: mailwoman was missing a pile of non-US coverage, and we were going to fix the parser to close the gap. By morning we'd shipped something better than planned, killed the original idea, and caught ourselves about to spend a GPU budget on a bug that lived in a symlink. Here's how a night of "fix the model" turned into "stop trusting your evals."

You can point a Nominatim client at Mailwoman now. Same endpoints — /search, /reverse, /status — same JSON shape, so your existing code doesn't change. The difference is what's behind it: no PostgreSQL, no osm2pgsql import that takes hours and tens of gigabytes, no server you can't ship inside an app. A geocoder that answers Nominatim's questions from a SQLite file.

We had just watched our geocoder beat Nominatim across the United States by fifteen points, and we were feeling good about ourselves. So we pointed the same benchmark at Europe expecting a victory lap. Europe handed us a double-digit loss instead.

That sat badly. Not because losing is shameful — Nominatim is the bar, it carries the whole planet on community-contributed data, and clearing it anywhere is the goal. It sat badly because we didn't understand it. We knew our parser wasn't ten points worse in Europe than in America. So what was the gap actually made of?

This is the answer, the two fixes, and — because we'd be kidding you otherwise — the parts the fixes didn't reach. For the European leg we added a third system to grade against: Pelias, by way of geocode.earth, the hosted Elasticsearch stack a lot of people reach for. It turns out to be the real bar, and we'll be honest about where it still beats us.

Our parser scores 28% on Spanish addresses. By the metric that reads its labels, it is, on paper, broken in a country of forty-seven million people.

That metric is measuring the wrong thing. The moment we measured the right one, the panic fell apart.

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.

We left the last postcode story with a promise and a bill. The promise was that the "which country is this" signal has to come from the trained model reading the whole string, because the postcode on its own settles the question less than half the time. The bill was that this is the expensive version of the feature. This is the post where we paid it: we built the country signal into the model, watched it do something great, and then watched it refuse, in the most instructive way we've hit all month, to do that same great thing in a different word order.

The great thing first, because you've earned it. We took the postcode's gazetteer membership, that [us, de, fr] answer from last time, and instead of handing it to a regex we injected it into the model at the postcode token itself. A small additive nudge on the hidden state, right where the five digits sit, carrying "here is what this code could be." On German addresses written the way Germans actually write them, it was worth thirty-five points of locality accuracy. It beat Pelias. For one evening we were heroes.

Then we looked at the international numbers and the floor gave way. Same model, same anchor, the same German cities, but now written house-number-first with the postcode trailing the city, the way our test feed renders them, and it scored a hair above a coin flip. The hero anchor was, on those rows, slightly worse than no anchor at all.

Three questions sit under the rest of this, so let me put them on the table before we start:

• When a parser "collapses" on a test, is the parser wrong, or is the test?
• Can you train one model to read an address in any order, or does each order cost you the other?
• And the one that took three retrains to answer: what does a learned anchor learn — the thing you asked for, or where you kept putting it?

We spent a good month teaching our resolver exactly one trick. Take a postcode, drop its centroid into the city polygon that happens to contain it, read off the city. It's a good trick. It got the Netherlands to 95% and Germany to 93%, and for a while it felt like the whole problem was going to fall to it. Then we pointed it at Japan, and Japan calmly informed us that it has no city polygons to drop anything into.

What follows is a two-country story about what a geocoder can still do when the map underneath it goes thin, and where it finally can't. Japan we resolved anyway, 94% of the way, by putting the polygon down and asking a different question. Korea handed the same problem back to us turned inside-out: it let us pin the coordinate perfectly, every time, and then stopped us cold at the one thing we were really after, which is the name of the place you've landed in.

Three questions sit under all of it, so let me put them on the table before we start:

• What do you do when the gazetteer gives you points where you expected shapes?
• Does the move that rescues Japan actually generalize, or did we get lucky once and dress it up as a method?
• And the question with no comfortable answer: what happens when the map is simply missing the part of a country you most need to see?

We set out to fix a small wart in our address parser and came away with a number that told us to put the screwdriver down.

Here is the wart. When our postcode extractor sees a five-digit run and wants to know whether it's a real postcode or just a house number that happens to look like one, it peeks at the words sitting next to it and checks them against every country's street vocabulary we know — American, German, French, all at once. That "all at once" is fine at three countries. At twenty it gets loud, and a German street suffix starts shadowing an English word by sheer coincidence. So we went looking for the clean way to tell the extractor which country's words to bother with.

That question has a much bigger sibling, and chasing the sibling is where the story is.