5 posts tagged with "Corpus pipeline"

Feed an address into a sequence labeler and part of it goes missing. The model tags the tokens it recognizes — house number, street, city — and the characters it doesn't understand fall on the floor. decodeAsJSON hands you back a tidy object with every field in its place, and nothing in that object tells you the model just shrugged at a fifth of the input.

So you ship that parser. It looks great on the addresses you tested it against. And you have no idea what it's dropping in production, because the output format was built to hide exactly that.

We spend a lot of energy distrusting numbers that look too good. A validation score that jumps, an accuracy that rounds up suspiciously close to 100 — we've been burned by those, so we poke at them. The number that says you failed gets a free pass. Of course it's right; who lies to make themselves look bad?

Our evals did, three times this week. One of them nearly talked us out of a model we should ship. One invented a coverage problem we don't have. And one had us writing "3.3 km" into a model card for a geocoder that puts most addresses within a hundred meters. Each time the fix was the same, and embarrassingly cheap: stop reading the summary row and pull the actual records the summary is averaging over.

Here's a number that should make you nervous: our validation macro-F1 climbed from 0.71 to 0.73 on a retrain we were ready to ship. Every instinct says that's a win. The aggregate went up; the model is better; cut the release.

It wasn't, and it took three probes to prove it. The same model that scored higher on the average had gotten worse at the most basic address there is — a town and a state. This is the story of how the average lied to us, and how we caught it.

Our failure taxonomy finally has a line at the very top: of everything wrong with the parser, boundary instability is the one worth fixing first. So we spent a night building the training data to fix it. And then we didn't retrain. Three separate gates each caught a step that looked completely reasonable and was wrong underneath, and that's the story worth telling, more than the shard ever was. Those gates are the only reason you can move fast on your worst weakness without shipping a fix that looks great on the headline and rots underneath.

Our training corpus is built from source: hundreds of millions of address rows, stitched out of eleven raw data sources, written to disk over the course of a long unattended night. Last night's build ran for two and a half hours and then died on a single row. If you've ever launched a long job before bed and woken up to a stack trace instead of an artifact, you know the specific flavor of that disappointment. The questions I want to answer here: how does one row out of nearly 700 million take down hours of compute? Why, after we fixed the thing that killed it, did it crash again on the exact same row? And what do you change once you've learned that a correct assertion and a safe one are different animals?