Skip to main content

Loaded 4535 golden entries Loading modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'... Running evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.... 500/4535 (2.5s) 1000/4535 (4.8s) 1500/4535 (7.1s) 2000/4535 (9.4s) 2500/4535 (11.7s) 3000/4535 (14.1s) 3500/4535 (16.4s) 4000/4535 (18.7s) 4500/4535 (21.1s)

Error Analysis Report

Golden setgolden setA hand-labeled evaluation dataset of US, French, and adversarial addresses used as ground truth for evals.: 4535 entries Modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': NeuralAddressClassifier Time: 21.2s

Summary​

MetricCountRate
Exact matchexact matchThe share of eval items whose every component is correct (compared per-span or per-token). Stricter than per-tag F1, which credits partial correctness.77117.0%
Missed entities2684—
Boundary errors3698—
Confused tags1474—
Hallucinated tags2381—

Missed entities (2684)​

  • 6220 SE Salmon St, Portland, OR 97215, USA — missing street_prefix="SE"
  • 6220 SE Salmon St, Portland, OR 97215, USA — missing street_suffix="St"
  • PO Box 123, Burlington, VT 05401 — missing po_boxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise.="PO BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. 123"
  • 123 Main St Apt 4B, Springfield, IL 62701 — missing unitunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise.="Apt 4B"
  • Main St & 5th Ave, Portland, OR — missing intersection_a="Main St"

Boundary errors (3698)​

  • 6220 SE Salmon St, Portland, OR 97215, USA — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "Salmon" got "SE Salmon St"
  • Main St & 5th Ave, Portland, OR — regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.: expected "OR" got "Portland"
  • 500 W 7th St Ste 1100, Cincinnati, OH 45203-1234 — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "7th" got "W 7th St Ste 1100"
  • 123 main st portland or 97214 — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "main st" got "main"
  • CULLEN INSULATION INC, POBOX 3211 FARGO ND 58108, FARGO, ND 58108 — venuevenueA named, non-address place — a business, building, park, or stadium. Mailwoman's free-text point-of-interest component, added as a Tier 2 fine label.: expected "CULLEN INSULATION INC" got "CULLEN"

Confused tags (1474)​

  • Vermont, USA — regionregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality.: expected "Vermont" got "USA"
  • 05770 VT DELONG LN — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "DELONG LN" got "VT"
  • ND 58368 6th St SW — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "6th St SW" got "ND"
  • ND 58701 Airview Dr — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "Airview Dr" got "ND"
  • WY 82601 Mountain Way — streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.: expected "Mountain Way" got "WY"

Hallucinated tags (2381)​

  • PO Box 123, Burlington, VT 05401 — hallucinated streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.="PO BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise. 123"
  • Main St & 5th Ave, Portland, OR — hallucinated dependent_locality="Main St"
  • 12345 Mt Tabor Park, Portland, OR 97215 — hallucinated streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.="Mt Tabor Park"
  • Vermont, USA — hallucinated dependent_locality="Vermont"
  • 05764 Finel Hollow Road, VT — hallucinated house_number="0576"