Skip to main content

Per-locale confidence calibration — en-us v4.0.0 (#368 L2)

A separate isotonic table per locale, vs the single global table. ECE is measured on each locale's held-out split under three regimes: raw softmax, the global table, the locale table.

localenaccuracyECE rawECE global-tableECE locale-table
NL2010.9850.17330.04660.0096
DE1970.8930.09320.08890.0411
US53150.9730.06740.00440.0037
FR7960.9660.06410.01380.0117

Where the locale-table column beats the global-table column, a single global table is leaving calibration error on the table for that locale (the OOD locales especially). A multi-locale model should ship one calibration table per locale, selected by the locale gate.