One row crashed our corpus build. Twice, on the same character.
· 9 min read
Our training corpus is built from source: hundreds of millions of address rows, stitched out of eleven raw data sources, written to disk over the course of a long unattended night. Last night's build ran for two and a half hours and then died on a single row. If you've ever launched a long job before bed and woken up to a stack trace instead of an artifact, you know the specific flavor of that disappointment. The questions I want to answer here: how does one row out of nearly 700 million take down hours of compute? Why, after we fixed the thing that killed it, did it crash again on the exact same row? And what do you change once you've learned that a correct assertion and a safe one are different animals?