Training environment

Audience

🧪 Operator documentation. For contributors provisioning GPU trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. environments. If you want to use Mailwoman, see Getting started.

:::

Training environment — playpen container setup

How to bring a fresh playpen container to the point where it can train a classifier or run a tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. build. Captures the gap that bit v0.5.0 Thread C-train.

What you start with

A playpen-mailwoman container has:

• Ubuntu 24.04, Node 24 + Yarn 4, the mailwoman repo cloned at /home/agent/workspace/mailwoman/
• Python 3.12 system interpreter, uv available
• /data/corpus/... mounted (corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. shardsshardA partial output file of the corpus build, written in Parquet format. The training pipeline streams shards row by row., ~30 GB)
• /data/models/... mounted (tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. weightsparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values., modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' checkpointscheckpointA saved snapshot of the model weights and optimizer state during training. Mailwoman saves a checkpoint periodically so training can resume after a GPU hang.)
• No GPU device passthrough
• No ~/training-venv

The first two facts together are the blocker. TokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. (A0, A1) needed neither — it's pure CPU SentencePieceSentencePieceA language-independent subword tokenizer that splits text into pieces using a unigram language model. Mailwoman uses a SentencePiece tokenizer with a 48,000-token vocabulary and byte-fallback, trained on address data rather than general text.. Classifier trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. (C-train) needs both.

Step 1 — GPU passthrough

From the host (this is operator-side, not in-container):

incus config device add <container> gpu gpu
incus config device add <container> kfd unix-char source=/dev/kfd

The first line passes through /dev/dri/card0 + /dev/dri/renderD128 (graphics + render nodes). The second adds /dev/kfd (AMD compute device — ROCm's entry point). Both are live changes; no container restart needed.

Verify in-container:

ls -la /dev/kfd /dev/dri/

Expect to see kfd, card0, renderD128 all present with crw-rw-... permissions.

Step 2 — Bootstrap ~/training-venv

The mailwoman corpus-python package splits dependencies — heavy ML deps (PyTorch, Transformersneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.', Datasets, ONNXONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime.) live under the [train] extras so tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses.-only work doesn't pull them. Make a dedicated venv:

uv venv ~/training-venv --python=3.12
source ~/training-venv/bin/activate

Step 2a — Install PyTorch with the ROCm wheel FIRST

Critical ordering: install the ROCm-built PyTorch wheel before pip install -e .[train]. The generic resolverresolverThe component that converts parsed address components (locality, region, postcode) into coordinates by looking them up in the gazetteer. The resolver ranks candidates by name match, population, and proximity, and returns the best-matching place with its centroid or polygon. picks CUDA wheels by default on x86_64 even though they won't run on AMD GPUs.

For the lab's Radeon 780M (gfx1103, RDNA 3 iGPU):

pip install torch --index-url https://download.pytorch.org/whl/rocm6.2

Download is ~3 GB. Takes 5-10 min on home upstream.

Step 2b — Install corpus-python + train extras

cd ~/workspace/mailwoman/corpus-python
pip install -e .[train]

This adds transformers>=4.41, datasets>=2.19, onnx>=1.16, onnxruntime>=1.18, tqdm on top of the always-installed sentencepiece, pyarrow, pyyaml, numpy. Another ~1 GB. ~3-5 min.

Step 3 — Verify GPU detected

python -c '
import torch
print("cuda:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("device:", torch.cuda.get_device_name(0))
    print("hip:", torch.version.hip)
'

Expected output on the lab hardware:

cuda: True
device: AMD Radeon Graphics
hip: 6.2.41133-XXXXXXXX

torch.cuda.is_available() returns True on AMD because PyTorch's ROCm build exposes the AMD GPU under the same torch.cuda.* API. The HIP version confirms you got the ROCm wheel and not the CUDA wheel.

Step 4 — Runtime override for gfx1103

The Radeon 780M reports itself as gfx1103 but ROCm's compute kernels target gfx1100. Set this env var before any trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. command:

export HSA_OVERRIDE_GFX_VERSION=11.0.0

Without the override, kernel launches will fail with cryptic GPU errors. Persist it in ~/.bashrc or prefix every trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. invocation.

Step 5 — Other env quirks

• hipBLASLt unsupported: PyTorch emits a UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas on every Linear layerlayerOne transformer block — attention plus a feed-forward network, with normalization and residual connections — applied to every position. Stacking layers lets the model build up richer representations; Mailwoman's encoder has 6. call. Harmless — the hipblas fallback works correctly on gfx1103. Future PyTorch versions may stop logging this.
• amdgpu.ids: No such file or directory: also harmless. The file is shipped with mesa-amdgpu-vulkan-drivers which isn't installed in the playpen base image. Nothing fails as a result.
• Math SDPA: per the lab GPU notes, set torch.backends.cuda.enable_flash_sdp(False) and use math SDPA backend if attentionattentionThe core mechanism inside a transformer encoder. Each token's representation is updated by looking at every other token, with learned weights deciding how much each one matters. kernels misbehave. The current corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.-python configs already do this.

Cost summary

Step	Time	Notes
GPU passthrough	< 1 min	Host-side, no container restart
uv venv	< 5 sec
ROCm PyTorch wheel	5-10 min	~3 GB download
pip install -e .[train]	3-5 min	~1 GB
Verify GPU	< 5 sec
Total	~15 min

Why this isn't pre-baked

Two reasons the mailwoman container template doesn't ship a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-venv:

1. Image size: torch + transformersneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' + onnxruntimeONNX (Open Neural Network Exchange). An open format for machine learning models that enables interoperability between training frameworks and inference runtimes. Mailwoman ships its trained model as an ONNX file so it can run in Node.js and the browser via onnxruntime. add ~5 GB to the container image. Multiplied across every playpen container (most of which never train), this is wasteful.
2. Use-case split: tokenizertokenizerThe component that converts a raw address string into a sequence of numeric token IDs the model can process. Mailwoman's tokenizer is a SentencePiece unigram model trained specifically on postal addresses. trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. and corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. utility scripts use only the lighter sentencepiece + pyarrow deps. They don't want or need torch. Splitting via [train] extras keeps the common case small.

The right time to bake it in is when classifier-trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. becomes a routine container spawn (multiple times per week). Until then the 15-min one-time setup is cheaper than the per-container image bloat.

See also

• VERDICT_SMOKES.md — what to run once the environment is up
• PHASE_2_training.md — the trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. loop's expected configuration
• tokenizer-a0-baseline.md — the lighter-weightparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. alternative path (CPU-only, no torch needed)