4 posts tagged with "Methodology"

Feed an address into a sequence labeler and part of it goes missing. The model tags the tokens it recognizes — house number, street, city — and the characters it doesn't understand fall on the floor. decodeAsJSON hands you back a tidy object with every field in its place, and nothing in that object tells you the model just shrugged at a fifth of the input.

So you ship that parser. It looks great on the addresses you tested it against. And you have no idea what it's dropping in production, because the output format was built to hide exactly that.

Type Vienna, Austria into our geocoder and, until this week, it answered with a confident set of coordinates: 39.32, −81.54. That's Vienna, West Virginia — population about ten thousand, a few miles up the Ohio River from Parkersburg. The capital of Austria is 7,500 kilometers and one ocean away, and the geocoder had no doubt whatsoever.

Our parser scores 28% on Spanish addresses. By the metric that reads its labels, it is, on paper, broken in a country of forty-seven million people.

That metric is measuring the wrong thing. The moment we measured the right one, the panic fell apart.

Our resolver scored 93.7% on the metric we'd been quoting for months. On the same addresses, its median answer was 326 kilometers from the truth.

Both numbers are correct. That's the awkward part.