Token
Language Models & ArchitectureThe basic unit of data a language model reads and writes, a word, part of a word, or character, into which text is broken before processing.
Inference throughput is commonly measured in tokens per second, and the HBM3e bandwidth of Eleveight AI's B300 GPUs is much of what determines how many tokens a model can generate each second.
Overview
A token is the chunk of text a language model actually operates on. Models do not read whole words or sentences directly; text is first split into tokens, which may be entire words, fragments of words, or individual characters, and each token is converted into numbers the model can process. Both the input a model reads and the output it produces are sequences of these tokens, generated one at a time.
How it works
A tokenizer breaks text into tokens according to a fixed vocabulary learned from data, balancing the number of tokens against how much meaning each carries. The model processes the input tokens and then generates its response token by token, each new token predicted from everything that came before. This sequential generation is why output speed is measured in tokens per second and why longer responses take proportionally longer to produce.
Why it matters
Tokens are the currency of language-model compute. Throughput, cost, and the length of context a model can handle are all measured in them, and generating each token requires passing the full model over the sequence so far, a memory-bandwidth-intensive operation. This is why bandwidth so directly governs how quickly a model responds, and why token throughput is a headline measure of inference performance.
Use cases
- Measuring inference throughput and cost
- Sizing context windows for applications
- Estimating workload requirements
- Benchmarking model serving performance