Self-supervised pre-training for the mailwoman encoder — experiment spec (2026-05-30)

Status: SPEC ONLY — not launched. This documents the design, objective choice, cost, and success criteria for adding a self-supervised pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. phasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). to the neural parser. The operator asked to spec it before committing GPU.

The problem it targets

The encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). is trained from scratch on labeled synthetic data only — there is no self-supervised pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.. Two diagnosed pathologies are the known signature of exactly that:

1. Overconfidence — 81% of the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s wrong predictions are emitted at ≥0.9 confidence. Hendrycks et al. (ICML 2019) show "networks trained from scratch exhibit overconfidence ... pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. improves modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' calibration" even when it doesn't change accuracy. Desai & Durrett (EMNLP 2020) found pre-trained transformersneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' have out-of-domain calibration error up to 3.5× lower than from-scratch baselines.
2. Format / delimiter over-reliance — the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' leans on comma cues and degrades on non-canonical input. Pre-trained transformersneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' show "substantially smaller" degradation under distribution shift than from-scratch modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (Hendrycks et al., arXiv 2004.06100).

Both point to the same missing ingredient. Two prior recipe cycles (v0.6.x held, v0.7.x calibration null) failed to move these because they tuned the labeled corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. mixture — not the thing that's actually absent.

The key property: pre-training adds ZERO bytes to the shipped model

This is a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-time phasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary)., not a runtime component. The lifecycle changes from one phasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). to two:

today:  labeled synthetic → [train encoder + CRF from scratch] → int8 ONNX (~25MB)

with:   Phase 1 (NEW): unlabeled address strings → [self-supervised pre-train the
                        SAME 6L/256h encoder] → encoder weights
        Phase 2:       labeled data + Phase-1 weights → [fine-tune encoder + CRF]
                        → int8 ONNX (~25MB, byte-identical architecture)

The exported artifact is the same size and shape — same 6 layerslayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6., 256 hidden, ~25MB int8. Pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. only changes the initialization the fine-tune starts from (a learned prior instead of random noise). For ELECTRA specifically, the extra "generator" network used during PhasephaseA milestone in the implementation plan (Foundation, Corpus, Training, Integration, and forward-looking phases). Distinct from stage (runtime pipeline) and tier (model vocabulary). 1 is discarded after pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. — only the discriminator (= our encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters).) is kept. Nothing new ships. It is fully reversible: if it doesn't help, we keep shipping the current modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'.

Objective choice: MLM-40% vs ELECTRA-RTD vs biphasic CLM→MLM

Two research passes (2026-05-30) converged. For a bidirectional tagger where the whole sequence is available at inferenceinferenceRunning the trained model on new input to get predictions, as opposed to training, which produces the model. In Mailwoman that means a small transformer encoder reads an address string and classifies every token — house number, street, locality, region, postcode, and the rest. A Who's On First gazetteer can feed soft location hints into the pass, but the model makes the final call on every label. Where a generative model writes text token by token, Mailwoman's output is a retrieval-augmented token classification: one label per input piece., the objective must be bidirectional.

Objective	Verdict for a 29M offline BIO tagger
ELECTRA replaced-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-detection (RTD)	Primary recommendation. Best-documented sample/compute efficiency at our exact scale — ELECTRA-Small (14M, ~our architecture: 12L/256h/4head/1024-FFNfeed-forward networkThe per-token transformation inside each transformer layer that follows attention: a small two-layer network applied independently at every position.) reached 79.9 GLUE on 1 GPU × 4 days, beating BERT-Small (75.1) and GPT at ~45× less compute. Lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. is over all tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. (denser signal than masked-only). RTD's discrimination objective is plausibly less overconfidence-inducing than tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-prediction.
MLM at ~40% mask rate	Safe runner-up. What every strong small NER encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). (BERT, RoBERTa, GLiNER's DeBERTa, NuNER) is built on; simpler than RTD's generator+discriminator. 40% (not the classic 15%) is optimal at small scale per Wettig et al. (EACL 2023): more predictions = better optimization. SpanBERT-style spanspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. masking is a close variant, well-matched to multi-tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. spansspanA contiguous range of characters or tokens in the input string, tagged with an address component type (street, locality, postcode, etc.). Parsed addresses are represented as collections of spans, possibly nested in a tree. (BIO).
Pure CLM (GPT-style, causal)	Wrong default for a tagger. BERT's own ablation: a left-to-right variant scored 77.8 vs 88.5 F1 on the tokentokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.-level SQuAD task. LLM2Vec confirms causal modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' must be converted back to bidirectional to encode well.
Biphasic CLM→MLM	Only if a generative headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. joins the roadmap. The ICLR-2026 controlled study (Gisserot-Boukhlef et al., arXiv 2507.00994, 15k+ runs) found CLM is more data-efficient and fine-tuning-stable, ties-or-slightly-beats MLM on token classificationsequence labelingA machine learning task where a model assigns a label to each element in a sequence. In Mailwoman, the sequence is the tokenized address string and the labels are address component types (street, locality, postcode, etc.). specifically, and that a CLM→MLM two-stagestageOne of the dataflow stages in the runtime pipeline (normalize, locale gate, kind classify, phrase group, token classify, sequence correct, reconcile, resolve). Distinct from tier (model vocabulary) and phase (plan milestone). protocol (≈25–50% CLM steps, then MLM) captures both. Caveat: the "CLM ≥ MLM on TC" result is contested by the paper's own author threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. as a possible weak-backbone (EuroBERT) artifact, and all their modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' are 210M–1B — none at our 29M scale.

The CLM question, answered directly

Is there a real case for CLM here? A partial, conditional one. Pure CLM is the wrong default — a tagger consumes the full sequence bidirectionally, and CLM trails on every task family except (contestedly) token classificationsequence labelingA machine learning task where a model assigns a label to each element in a sequence. In Mailwoman, the sequence is the tokenized address string and the labels are address component types (street, locality, postcode, etc.).. The only scenario where CLM earns a place is if mailwoman later wants a generative repair / rewrite / canonicalizationcanonicalizationMapping a span to its canonical value: 'St' and 'Street' both become the USPS suffix ST; 'USA' and 'United States' both become ISO US. The operation that handles synonymy. Canonical values come from @mailwoman/codex, not a hand-grown alias list. headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. — the AddrLLM pattern (JD Logistics combined address parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. + rewriting in one modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', cutting parcelparcelA property polygon or record carrying a situs (site) address and often a separate owner mailing address. County GIS parcel aggregations are a training source for address-point variety and situs-vs-owner divergence. re-routing ~43%). If that lands on the roadmap, adopt the biphasic CLM→MLM recipe rather than pure CLM: early CLM data-efficiency + a bidirectional MLM/RTD finish. For pure BIO taggingBIO tagging (Begin-Inside-Outside). A token-level labeling scheme where each token is tagged as B-X (beginning of an entity of type X), I-X (inside an entity of type X), or O (outside any entity). Mailwoman uses BIO over SentencePiece tokens to annotate address components. today, that's speculative future-proofing, not a current need.

Note on calibration specifically: no study isolates CLM-vs-MLM downstream calibration — the calibration win is from pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. presence, not the objective. The objective-independent calibration lever is label smoothinglabel smoothingA training regularization technique: instead of putting 1.0 probability on the correct label, train to put 1 − ε on it and ε/(N−1) on each wrong label. Improves calibration; disabled in some Mailwoman runs for stability. + temperature scaling (already partially explored in v0.7.0). So: pre-train (any bidirectional objective) and keep label smoothinglabel smoothingA training regularization technique: instead of putting 1.0 probability on the correct label, train to put 1 − ε on it and ε/(N−1) on each wrong label. Improves calibration; disabled in some Mailwoman runs for stability. in the fine-tune.

The experiment (cheapest-signal-first)

Stage A — TAPT probe (lowest risk, do first). Take the address strings we already template, strip the labelscomponent tagOne of the 33 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag., and run whole-word MLM at ~40% on the existing 29M architecture + the frozen existing 16k SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text. tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. for a few hours on one A100 — even just the ~10M strings we can cheaply assemble. Then fine-tune the BIO headattention headOne of several parallel attention computations in a layer, each free to focus on a different kind of relationship between tokens. Their outputs are concatenated — 'multi-head attention'. Mailwoman uses 4 heads. from those weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. with a low LR + label smoothinglabel smoothingA training regularization technique: instead of putting 1.0 probability on the correct label, train to put 1 − ε on it and ε/(N−1) on each wrong label. Improves calibration; disabled in some Mailwoman runs for stability. (ideally keeping a small auxiliary MLM lossloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. to resist catastrophic forgetting, which otherwise erases the calibration benefit — Tao et al. arXiv 2305.19249).

Why from-scratch domain MLM and not continue-pretraining a general modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.': addresses are vocabularyvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it.-distinct and terse; PubMedBERT (arXiv 2007.15779) showed from-scratch + in-domain vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. beats continued general-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' pretraining for a narrow domain — and it lets us keep our existing custom 16k vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. and tiny size.

Stage B — scale to RTD (only if Stage A moves the needles). ELECTRA-RTD pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. on 50–100M strings, still ~1 GPU-day.

Recipe (Stage A)

• Mask rate ~40% (small-modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' optimum; not the classic 15%).
• Seq len 128 (matches the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s max); AdamWoptimizerThe component that decides how to update parameters from the gradient — Adam/AdamW being the common choice, adding momentum and per-parameter scaling on top of plain gradient descent.; peak LR ~5e-4, ~6% warmupwarmupThe early phase of training where the learning rate ramps up from 0 to its peak value before cosine decay. Mailwoman uses a linear warmup., linear decay; large batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU..
• TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. frozen and shared between pre-train and fine-tune — a vocabvocabularyThe fixed set of tokens a tokenizer can produce. Mailwoman's SentencePiece vocabulary is tens of thousands of subword pieces, with byte fallback for anything outside it. swap invalidates transfer. (This is the single most important do-not-break rule.)
• Fine-tune from the pre-trained weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. with low LR + label smoothinglabel smoothingA training regularization technique: instead of putting 1.0 probability on the correct label, train to put 1 − ε on it and ε/(N−1) on each wrong label. Improves calibration; disabled in some Mailwoman runs for stability..

Cost

• 1 GPU-day on ModalModalA cloud GPU platform (modal.com) where Mailwoman trains its neural models on NVIDIA A100 GPUs. Training runs are launched via scripts/modal/train_remote.py and typically complete in ~1 hour. A100, **$3–5**. MosaicBERT-Base (137M) reached 79.6 GLUE in 1.1h on 8×A100 ($22); a 29M modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' on a narrow corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. is far cheaper.

Success criteria (compare pre-trained-then-fine-tuned vs from-scratch baseline)

1. Calibration — fraction-of-wrong-predictions-at-≥0.9 (the 81% number) and ECEECE (Expected Calibration Error). A metric that measures how well a model's confidence scores align with its actual accuracy. Lower is better. Mailwoman's held-out ECE drops from 0.067 (raw) to 0.0035 (calibrated).. Primary — this is the pathology we're attacking.
2. OOD robustness — accuracy on a comma-stripped / non-canonical held-out set.
3. ResolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. end-to-end Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. — the product metric (must not regress; ideally improves via fewer confident-wrong parsesaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates.).
4. Harness pass-rate as a regression gate (not the promote bar — see the harness-lineage doc).

Pre-registered revert

If Stage A does not improve calibration (the ≥0.9-wrong fraction) and resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. Acc@1Acc@1 (accuracy at 1). The share of eval rows whose top-ranked resolver candidate is the correct place, regardless of coordinate distance — 'did we pick the right place', independent of geocode precision. doesn't move ≥1.5pp, stop — the from-scratch modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' stands and we've learned pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. isn't the lever at this scale.

Interleaving with the resolver-depth track

Parallel — touches zero resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. code. The resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. consumes the parser's BIO output; pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. changes only the encoderencoderThe part of a transformer that turns input tokens into contextualized vector representations. Mailwoman's classifier is a small encoder-only transformer (~30M parameters). weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. behind that interface. Convergence point: land the OpenAddressesOpenAddresses (OA). A global open aggregation of address points collected from many official sources. A primary source of component-supervised training data outside proprietary registries. real-point evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. first so both tracks share one honest scoreboard, and re-run the arbiter evalevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. after any new modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' (a better-calibrated parser may change the arbiter's win margin).

Sources

• Hendrycks et al., "Using Pre-Training Can Improve Model Robustness and Uncertainty" (ICML 2019) — arXiv 1901.09960
• Desai & Durrett, "Calibration of Pre-trained Transformersneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'" (EMNLP 2020) — arXiv 2003.07892 / aclanthology 2020.emnlp-main.21
• Hendrycks et al., "Pretrained Transformers Improve Out-of-Distribution Robustness" (ACL 2020) — arXiv 2004.06100
• Clark et al., "ELECTRA" (ICLR 2020) — arXiv 2003.10555
• Wettig et al., "Should You Mask 15% in MLM?" (EACL 2023) — aclanthology 2023.eacl-main.217
• Gisserot-Boukhlef et al., "Should We Still Pretrain Encoders with MLM?" (ICLR 2026) — arXiv 2507.00994
• Gururangan et al., "Don't Stop Pretraining" (DAPT/TAPT, ACL 2020) — aclanthology 2020.acl-main.740
• Gu et al., "PubMedBERT" / domain-specific from-scratch (2020) — arXiv 2007.15779
• Tao et al., calibration-preserving fine-tuning — arXiv 2305.19249
• G2PTL (Cainiao, address pre-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input., MLM+HTC+geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available.) — arXiv 2304.01559; GeoBERT — MDPI Applied Sciences 12/24/12942; deepparse — arXiv 2006.16152
• AddrLLM (generative address rewriting, JD Logistics) — arXiv 2411.13584