Skip to main content

Phase 6 — Japan Expansion (Architecture Validation)

Goal: add Japanese address support. This is the validation milestone for the architecture: if the schema, classifier interface, and policy system survive JP without core refactor, the design is sound.

Status: deferred. Begin only after PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 3 has shipped, StagesstageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). 2 and 3 of trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. have completed for US/FR, and the human has confirmed.

This document is a sketch. Detailed plan written when PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 6 begins.

Why JP specifically​

  • Structurally different addressing. JP has no streetsstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels.. Addresses are nested blocks (chōmechōmeIn Japanese addressing, a district-level subdivision in the block-based chōme / banchi / gō numbering scheme, which uses area-and-block numbers instead of street names./banchibanchiIn Japanese addressing, a block within a chōme. The middle level of the chōme / banchi / gō hierarchy./go). If any core code assumes "streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. is mandatory," it breaks here.
  • Different script. CJK characters expose tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. limits. SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. with byte_fallback=true (set in PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1) should handle this, but real test only happens with real JP data.
  • Excellent open data. MLIT (国土交通省) publishes comprehensive address data freely. Quality comparable to BANBAN (Base Adresse Nationale). France's authoritative open national address register — the highest-quality training source for French addresses, with full component structure..
  • Existing reference implementation. japanese-address-parser Rust crate is a baseline to compare against.

What must already work before Phase 6​

  • Schema has JP-specific tags reserved (it does — see reference/SCHEMA.md)
  • LocaleProfile.componentsSupported is honored everywhere (it must be by PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 0)
  • No hardcoded assumption that street or house_number is required anywhere
  • TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. was trained with sufficient character coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. to handle CJK (set in PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1)

If any of these aren't true, the design failed and the project needs a PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 0 revisit.

Sketch tasks​

Data​

  • Adapter for MLIT address data (CSV format, comprehensive coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. of JP)
  • Adapter for WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. JP admin records (already accessible)
  • Synthesis rules for JP variants: kanji vs hiragana vs romaji, traditional vs modern numbering

Tokenizer​

  • Validate existing tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.'s CJK handling. May need to retrain with more JP samples in the SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. mix.
  • If retraining, that's a corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. version bump. Don't ship a tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. change in a patch release.

Model​

  • Continue trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. from US/FR checkpointcheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang. with JP data added
  • Or train a separate JP-only modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — depends on whether shared vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. benefits transfer or hurts via interference
  • Decision point: measure both, pick winner

Schema​

  • Add JP-specific LocaleProfile with componentsSupported including JP tags, excluding street and house_number
  • Register JP rule classifiers (minimal — most work is the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.')

Validation​

  • Compare against japanese-address-parser on held-out JP addresses
  • If we underperform a single-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. specialist, that's expected and acceptable in v1
  • If we underperform by > 10 F1 points, investigate — likely a vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. or alignmentalignmentThe step in the corpus pipeline that takes a (raw, components) pair from an adapter and produces a (raw, tokens, BIO labels) row by finding each component's text inside the raw string and labeling the matching tokens. issue

Open questions for when Phase 6 begins​

  • Single multilingual modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' vs per-localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'?
  • Romaji-only support, kanji-only, or both?
  • How aggressively to normalizenormalizeStage 1 of the runtime pipeline: deterministic input preprocessing (Unicode NFC, punctuation normalization, whitespace collapse). Returns a NormalizedInput with an offsetMap that maps normalized positions back to the raw input. variants (full-width vs half-width digits, traditional vs simplified kanji where applicable)
  • Whether to include Korean/Chinese as bonus localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. since vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. will be CJK-friendly by then

What success looks like​

When you can run:

npx mailwoman parse --locale ja-JP "東京都千代田区丸の内1-1-1"

and get a parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. with prefecture: 東京都, municipality: 千代田区, district: 丸の内, block: 1, sub_block: 1, building_number: 1 — and no core code changed since PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 3, only adapters, weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., and localelocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. profiles — the architecture is validated.

If core code did need to change, the change is the deliverable, not the JP support. Document what changed and why so future localeslocaleThe combination of language and country an address comes from. en-US and fr-FR are the locales Mailwoman ships weights for. (Korean, Thai, etc.) benefit.