Skip to main content

DeepSeek — max_tokens covers reasoning, not just output

A short article on one DeepSeek API quirk that has burned every mailwoman threadthreadA parallel workstream within a release. Threads compose; they are not sequential milestones like phases. that called it with reasoning enabled. Worth knowing before you write the next one.

The trap​

DeepSeek's chat-completions API takes a max_tokens parameterparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values.. The natural reading is "stop after this many output tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.." The actual semantic is "stop after this many reasoning + output tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words.." With reasoning_effort=low, the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s internal reasoning happily consumes 75% of the budget before any output is emitted.

Thread B2's first run was configured with max_tokens=20000 and reasoning_effort=low. The modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s reasoning ate ~15K tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. per response, leaving ~5K for output. Most transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. responses needed more than 5K (multi-script transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. of an address row plus its annotation is large). Every truncated response was returned with finish_reason=length and a partial body.

If the worker treats finish_reason=length as a successful completion (most starter implementations do), the partial body is fed into downstream parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. and the row is silently malformed. Throughput appears fine — the worker is consuming responses at the API's nominal rate — but the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. that lands on disk is partial.

How to detect it​

The single best signal is the rejection-pipelinestaged pipelineMailwoman's runtime architecture: a sequence of pure-function stages (normalize → query-shape → locale-gate → kind-classifier → phrase-grouper → classifier → decoder) connected by typed handoffs. Each stage is published as its own npm package. log. If the substring validator (see Synthetic corpus — alignment validation) starts rejecting rows in batchesbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU., with reasons like not-in-raw:postcode clustered at end-of-row positions, the LLM responses are being truncated. The validator is doing the right thing — it would silently corrupt the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. without it.

The second signal is finish_reason itself. Logging the distribution per batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. shows the truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. rate directly. A healthy run is finish_reason=stop >99% of the time; truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. rates above 1% are a configuration bug.

The fix​

Three changes to the worker:

  1. Bump max_tokens generously enough that reasoning + output fits. For mailwoman's address-transliterationtransliterationConverting a name from one writing system to another while preserving pronunciation (Cyrillic → Latin, for instance). Needed for multilingual address handling and corpus synthesis. prompts, B2 settled on max_tokens=60000. Pure-output tasks can stay lower (~5K) but anything with reasoning should err well above what feels comfortable.
  2. Retry on finish_reason=length rather than accepting the partial response. The B2 patch added a one-shot retry with max_tokens bumped to 90K (1.5× the configured value). If the retry also truncates, the row is dropped with a structured rejection reason — the validator never sees malformed input.
  3. Log truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. rate per batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU.. Surfaces the problem before it can poison the corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. The B2 worker writes a per-batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. summary line; the truncated= counter is the headline.

Why the API does it this way​

Reasoning modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' are still pricing in compute differently than vanilla generation. Reasoning tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. cost the same as output tokenstokenOne word or subword in the tokenized input. For the neural classifier, tokens come from SentencePiece (subword units); for the rule classifiers, tokens are whitespace- and punctuation-separated words. but are not returned to the caller. Charging them against the same budget is the simple modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' — one number, predictable cost ceiling. The trap is that the parameterparameterA single learned number inside a model — one weight or bias. Mailwoman's encoder has roughly 30 million of them; training is the search for good values. name does not advertise the semantic.

DeepSeek's documentation does stateregionThe first-level administrative subdivision of a country — a US state, a French region, a province. The component between country and locality. this, in a sentence buried under the reasoning-effort overview. If you discover the trap by reading the docs, well done. If you discover it by watching your corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic shards (boundary stress, order variants, negative space). Managed by @mailwoman/corpus. get silently truncated, you are in good company — Thread B2's first run was the first time we hit it inside the playpen workflow, and the patch took longer than the original generation would have if we had set the budget right at the start.

Carry forward​

When writing the next DeepSeek-driven worker:

  1. Default max_tokens to 3× the natural output ceiling, not 1×.
  2. Wire finish_reason=length as a retry, never a success.
  3. Surface truncationtruncationCutting an input down to the max sequence length (or an LLM response to its token limit), discarding everything past the cap. rate in the per-batchbatch sizeHow many examples the model processes before each parameter update. Larger batches give smoother gradients but cost more memory; gradient accumulation simulates a big batch on a small GPU. summary line.
  4. Run a 50-row smoke pass first and verify finish_reason=stop rate before committing to the full run.

See also​