Day 03: Why tokenization quietly affects cost, limits, and reliability
How a low-level text detail quietly becomes a product constraint for cost, latency, and reliability.
Tokenisation is the most boring topic in this series and it will show up on your invoice.
Models do not read characters or words. They read tokens, chunks of text carved up by a scheme optimised for the training data, which means English prose is efficient and almost everything else is not.
The efficiency gap nobody budgets for
Same meaning, wildly different token counts:
Plain English is roughly four characters per token. Code is worse: indentation, symbols, and camelCase identifiers fragment aggressively. JSON is worse still, because every brace, quote, and repeated key is paid for on every record. Many non-English languages are dramatically worse; the same sentence can cost two or three times more in one language than another, which quietly makes your product more expensive for some users than others.
Then there are the inputs nobody tests: a PDF extraction full of stray whitespace, a table flattened into repeated delimiters, a log file, a base64 blob that slipped into a document.
Your demo prompt exercised none of this.
Two failures, one cause
The bill. Costs modelled on sample prompts and encountered on real inputs. The gap is usually a multiple, not a percentage, and it appears at the worst moment, when usage grows.
Silent truncation. This is the sharper one. A long input exceeds the context limit, something upstream trims it, and the model answers confidently from the fraction that survived. No error. No warning. Just an answer that is missing the part of the document that mattered, and no signal distinguishing it from a good answer.
Neither is fixed by better prompting. Both need measurement, budgets, and normalisation before the expensive call.
What to do
Measure token usage on your worst realistic input, not a clean example: the biggest PDF, the messiest table, the longest conversation, the non-English case. That number is your real per-request budget.
Then normalise noisy text before it hits the model. Stripping duplicated whitespace and boilerplate from extracted documents is unglamorous and routinely cuts token counts by a meaningful fraction for free.
And make it visible. Token cost per request, logged and dashboarded from day one. Design conversations get concrete very quickly once everyone can see what a feature costs.
Closing thought
Tokens are where product behaviour, cost, and infrastructure meet. It is a low-level detail right up to the moment it becomes a budget conversation with your CFO.
Day 3 of 60 Days of Production AI Systems.
Previous: Day 02: Why LLMs feel intelligent but still generate one step at a time
Next: Day 04: Why more context can make an AI system worse
Day 3 of 60 · AI Systems Basics (chapter 1 of 10)
