Articles about specific evaluation rigs, test sets, and harnesses built to measure model or resolver correctness.
View All Tags
Last night the plan was straightforward: mailwoman was missing a pile of non-US coverage, and we were going to fix the parser to close the gap. By morning we'd shipped something better than planned, killed the original idea, and caught ourselves about to spend a GPU budget on a bug that lived in a symlink. Here's how a night of "fix the model" turned into "stop trusting your evals."
We had just watched our geocoder beat Nominatim across the United States by fifteen points, and we were feeling good about ourselves. So we pointed the same benchmark at Europe expecting a victory lap. Europe handed us a double-digit loss instead.
That sat badly. Not because losing is shameful — Nominatim is the bar, it carries the whole planet on community-contributed data, and clearing it anywhere is the goal. It sat badly because we didn't understand it. We knew our parser wasn't ten points worse in Europe than in America. So what was the gap actually made of?
This is the answer, the two fixes, and — because we'd be kidding you otherwise — the parts the fixes didn't reach. For the European leg we added a third system to grade against: Pelias, by way of geocode.earth, the hosted Elasticsearch stack a lot of people reach for. It turns out to be the real bar, and we'll be honest about where it still beats us.
Our parser scores 28% on Spanish addresses. By the metric that reads its labels, it is, on paper, broken in a country of forty-seven million people.
That metric is measuring the wrong thing. The moment we measured the right one, the panic fell apart.
Mailwoman ships two address parsers in the same box.
The first, which we call v0, is our TypeScript port of the Pelias parser — a rules engine: tokenize, classify each token against dictionaries and patterns, solve for the most plausible arrangement under a pile of hand-written constraints. It is fast, deterministic, and the product of years of accumulated postal wisdom. It is also the thing we set out to beat.
The second is the neural classifier — a sequence labeler trained on a BIO-tagged corpus, the one this blog has spent most of its life arguing with.
So: a year in, did the neural net actually beat the rules parser? The answer is mostly a tie, until the address gets weird — and the weird is where it gets interesting.
Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.
Both numbers are correct. That's the awkward part.