Two days ago we retired a piece of decode-time machinery we'd been leaning on, and it exposed something the crutch had been hiding: our parser can't read a French address backwards. Last night we tried to teach it to. We got most of the way, hit a wall at 87%, and did the obvious thing and turned the training signal up, fully expecting the last few points to fall out. They didn't. The model got worse, and the way it got worse is the whole story. The questions on the table: can a pile of reordered examples teach a model where a house number lives? When that pile gets you 80% of the way, can you just add more? And what does the failure tell you about the thing you were actually training?

Our training corpus is built from source: hundreds of millions of address rows, stitched out of eleven raw data sources, written to disk over the course of a long unattended night. Last night's build ran for two and a half hours and then died on a single row. If you've ever launched a long job before bed and woken up to a stack trace instead of an artifact, you know the specific flavor of that disappointment. The questions I want to answer here: how does one row out of nearly 700 million take down hours of compute? Why, after we fixed the thing that killed it, did it crash again on the exact same row? And what do you change once you've learned that a correct assertion and a safe one are different animals?

Last night our release gate failed twice before it passed, and both failures turned out to be the same lesson wearing different hats. If you've ever trained a model, watched it flunk an eval, and reached for more data or more parameters: this post is about the third option you might be skipping. The questions on the table: why did a model that scored 89 on post-office boxes in validation score 60 at the gate? Why did fixing that break French postcodes? And what does any of this say about where a parser's knowledge lives?

Yesterday we wrote about a lookup table that scored a perfect 100 and nearly talked us out of our own architecture. We ended that post with a rule: write the bar down before you look at the score. What we didn't know was that the next 24 hours would test that rule four separate times, and that the fourth test would flip a conclusion we'd been quoting for two weeks.

This is the story of shipping v4.2.0: a consolidation, a bar that lowered itself, a consultant who was confidently wrong twice, a capacity wall with a number on it, and a benchmark harness that had been starving our own model for four releases.

This morning we published a post that ended with a tidy rule: some address tags don't want a neural network, they want a lookup table. Country names are a closed list in a known position. Our deterministic matcher scored a perfect 100 on the eval. The retrained model scored a mess. Case closed, we wrote.

By the afternoon we'd reopened the case, and the verdict flipped — hard enough that we've retracted the morning post rather than leave the wrong conclusion lying around for someone to cite. This is the story of how a perfect score nearly talked us out of the entire premise of the project.

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.

There is a particular kind of engineering misery where you fix a bug three times and it never gets better, because the bug is in your ruler. This is that story.

Our neural parser handles German two ways. Native order — Hauptstraße 5, 10115 Berlin — is the layout real German feeds and real German people use. International order — 5 Hauptstraße, Berlin, 10115 — is the Americanized layout our evaluation set happens to ship. For months, international-order German "collapsed": locality accuracy sat around 44% while native cleared 80%. We had a story for it. The postcode anchor — a side-channel that feeds the model a country hint derived from the postcode — sits at the trailing postcode, which in international order lands on the far side of the locality from where it's needed. Plausible. So we retrained.

Ask a geocoder for "Berlin" and it has to make a choice. There's the one in Germany, obviously. There's also Berlin, New Hampshire (population nine thousand and change), Berlin, Wisconsin, Berlin, Connecticut, and a dozen more scattered across the United States like the name was on sale. The parser hands you the word Berlin tagged as a locality; something downstream has to decide which dot on the map that is. How would you even know if it picked right?

For a long time our answer was a scorecard that checked the name. Did the resolved place's name equal the expected name? Tick. Move on. It is a completely reasonable thing to measure, and it was lying to us for months.

We left the last postcode story with a promise and a bill. The promise was that the "which country is this" signal has to come from the trained model reading the whole string, because the postcode on its own settles the question less than half the time. The bill was that this is the expensive version of the feature. This is the post where we paid it: we built the country signal into the model, watched it do something great, and then watched it refuse, in the most instructive way we've hit all month, to do that same great thing in a different word order.

The great thing first, because you've earned it. We took the postcode's gazetteer membership, that [us, de, fr] answer from last time, and instead of handing it to a regex we injected it into the model at the postcode token itself. A small additive nudge on the hidden state, right where the five digits sit, carrying "here is what this code could be." On German addresses written the way Germans actually write them, it was worth thirty-five points of locality accuracy. It beat Pelias. For one evening we were heroes.

Then we looked at the international numbers and the floor gave way. Same model, same anchor, the same German cities, but now written house-number-first with the postcode trailing the city, the way our test feed renders them, and it scored a hair above a coin flip. The hero anchor was, on those rows, slightly worse than no anchor at all.

Three questions sit under the rest of this, so let me put them on the table before we start:

• When a parser "collapses" on a test, is the parser wrong, or is the test?
• Can you train one model to read an address in any order, or does each order cost you the other?
• And the one that took three retrains to answer: what does a learned anchor learn — the thing you asked for, or where you kept putting it?

We spent a good month teaching our resolver exactly one trick. Take a postcode, drop its centroid into the city polygon that happens to contain it, read off the city. It's a good trick. It got the Netherlands to 95% and Germany to 93%, and for a while it felt like the whole problem was going to fall to it. Then we pointed it at Japan, and Japan calmly informed us that it has no city polygons to drop anything into.

What follows is a two-country story about what a geocoder can still do when the map underneath it goes thin, and where it finally can't. Japan we resolved anyway, 94% of the way, by putting the polygon down and asking a different question. Korea handed the same problem back to us turned inside-out: it let us pin the coordinate perfectly, every time, and then stopped us cold at the one thing we were really after, which is the name of the place you've landed in.

Three questions sit under all of it, so let me put them on the table before we start:

• What do you do when the gazetteer gives you points where you expected shapes?
• Does the move that rescues Japan actually generalize, or did we get lucky once and dress it up as a method?
• And the question with no comfortable answer: what happens when the map is simply missing the part of a country you most need to see?