How Large Language Models Actually Work: Tokens, Transformers and Attention Explained

Large language models feel like magic until you see the machinery. Under the hood, an LLM does one thing: given a sequence of tokens, it predicts a probability distribution over the next token. Everything else — chat, coding, summarization, "reasoning" — is built on repeating that single step very well.

This guide walks through that machinery in the order data actually flows through it: text becomes tokens, tokens become vectors, vectors pass through transformer layers, and a probability distribution comes out the other end. Along the way we'll cover training, inference, context windows and why models confidently make things up.

#Key takeaways

• An LLM is a next-token predictor. Generation is a loop: predict one token, append it, repeat.
• Text is split into tokens (sub-word pieces), not words or characters. Pricing, context limits and some weird failure modes all come from this.
• The transformer architecture uses self-attention so every token can look at every other token in the context.
• Models learn in stages: pre-training on huge text corpora, then instruction tuning and preference tuning to behave like assistants.
• Hallucinations are a natural side effect of predicting plausible text, which is why grounding techniques like RAG matter.

#Step 1: Tokenization — turning text into numbers

Neural networks operate on numbers, so the first job is converting text into integers. LLMs use sub-word tokenizers (Byte-Pair Encoding and its variants are the most common). A tokenizer learns a vocabulary of frequent character sequences from training data, typically somewhere between 30,000 and a few hundred thousand entries.

Common words become a single token. Rare words get split into pieces:

text"The server is overloaded"   → ["The", " server", " is", " overloaded"]
"Qyvera's tokenizer"         → ["Q", "y", "vera", "'s", " token", "izer"]

A few practical consequences fall straight out of this:

• Pricing and limits are counted in tokens, not words. In English, one token averages roughly three-quarters of a word, but code, non-English text and unusual strings use more tokens per character.
• Character-level tasks are awkward. Counting letters in a word or reversing a string is hard when the model never "sees" individual characters.
• Numbers are tokenized inconsistently, which is one reason arithmetic without tools is unreliable.

#Step 2: Embeddings — tokens become vectors

Each token ID is mapped to a learned vector, its embedding, often with thousands of dimensions. These vectors start random and are adjusted during training so that tokens used in similar contexts end up near each other in that high-dimensional space.

Because transformers process tokens in parallel rather than sequentially, the model also needs to know where each token sits. Positional information is added — modern models commonly use rotary position embeddings (RoPE), which encode relative position directly into the attention computation.

If you want to go deeper on how these vectors power search and retrieval, see text embeddings explained.

#Step 3: The transformer and self-attention

The transformer, introduced in the 2017 paper "Attention Is All You Need", is a stack of identical blocks. Each block has two main parts:

1. Multi-head self-attention — lets each token gather information from other tokens in the context.
2. A feed-forward network (MLP) — transforms each token's representation independently. A large share of the model's parameters, and much of its stored "knowledge", lives here.

Residual connections and normalization wrap both parts so that very deep stacks (dozens to over a hundred layers) remain trainable.

#How attention works, intuitively

For each token, the model computes three vectors from its current representation:

Attention scores are the dot products of one token's query with every other token's key, scaled and passed through a softmax so they sum to 1. The output for that token is the weighted sum of all the values:

textAttention(Q, K, V) = softmax(Q · Kᵀ / √d) · V

In a decoder-only model (the GPT-style architecture most chat LLMs use), a causal mask stops tokens from attending to positions after them, so the model can't cheat by looking at the answer it's supposed to predict.

Multi-head attention runs several of these in parallel with different learned projections. In practice, different heads pick up different patterns: some track syntax, some link pronouns to their subjects, some copy tokens from earlier in the context.

#Why attention was such a big deal

Earlier architectures (RNNs, LSTMs) processed text one token at a time, which made them slow to train and bad at long-range dependencies. Attention connects any two positions in a single step and parallelizes well on GPUs. That combination is what made training on trillions of tokens practical.

The cost is that naive attention scales quadratically with sequence length: doubling the context quadruples the attention computation. Much of the engineering behind long context windows, including FlashAttention, grouped-query attention and sliding-window variants, exists to manage that cost.

#Step 4: From vectors back to a prediction

After the final transformer layer, the representation of the last token is projected back onto the vocabulary, producing one score (a logit) per possible next token. A softmax turns those into probabilities.

The model then samples a token from that distribution. Sampling settings control how adventurous it is:

• Temperature sharpens or flattens the distribution. Low values (0–0.3) make output more deterministic; higher values add variety.
• Top-p (nucleus) sampling keeps only the smallest set of tokens whose cumulative probability reaches p.
• Top-k keeps only the k most likely tokens.

The chosen token is appended to the input and the whole process repeats. This loop is why output streams word by word, and why generation speed is measured in tokens per second.

#How LLMs are trained

#Pre-training

The model is trained on a very large corpus of text and code with one objective: predict the next token. Every position in every document becomes a training example, and the model's weights are nudged by gradient descent to reduce prediction error. To get good at predicting text, the model is forced to absorb grammar, facts, reasoning patterns and coding conventions, because all of those help prediction.

#Instruction tuning and preference tuning

A pre-trained model completes text; it doesn't follow instructions well. Two further stages make it useful:

1. Supervised fine-tuning (SFT) on curated examples of instructions and good responses.
2. Preference optimization, such as reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), or AI-feedback variants. Here the model learns from comparisons of better and worse answers.

Newer "reasoning" models add training that rewards working through problems step by step before answering, often with verifiable rewards on math and code.

#Inference: what happens when you call an API

When you send a prompt, the provider runs two phases:

1. Prefill — the whole prompt is processed in parallel, and intermediate attention results are stored in a KV cache.
2. Decode — tokens are generated one at a time, reusing the KV cache so earlier tokens aren't recomputed.

This explains two things developers notice. Long prompts increase time-to-first-token, because prefill does more work. And output tokens usually cost more than input tokens, because decoding is sequential and harder to batch efficiently. Techniques like prompt caching exploit the KV cache so that a repeated prefix doesn't pay the full prefill cost twice. We cover the cost side in LLM cost optimization.

#The context window

The context window is the maximum number of tokens the model can attend to at once, prompt plus output. Windows have grown from a few thousand tokens to hundreds of thousands and beyond, but bigger isn't automatically better:

• Cost and latency grow with the amount of context you send.
• Models can under-weight information buried in the middle of very long contexts.
• Relevant, well-ordered context almost always beats "dump everything in".

#Why LLMs hallucinate

A model trained to produce plausible continuations will sometimes produce plausible-sounding falsehoods. It has no built-in database lookup and no native sense of "I don't know". Hallucinations get more likely when:

• The question concerns rare facts, recent events or private data the model never saw.
• The prompt implies an answer exists (for example, "cite three papers that prove…").
• Sampling temperature is high.

The fixes are architectural rather than magical: ground the model in retrieved documents, constrain output formats, let it call tools for facts and math, and evaluate systematically. See how to reduce LLM hallucinations for a production checklist.

#What this means when you build with LLMs

Understanding the mechanics changes how you design features:

• Put the most important instructions and data where the model will use them, and keep prompts focused.
• Use tools for exact work (math, lookups, dates) instead of hoping the model gets it right.
• Stream responses to hide decode latency in user-facing apps.
• Cache stable prompt prefixes such as system prompts and reference documents.
• Measure, don't guess. Build an eval set before you switch models or rewrite prompts. Our guide to evaluating LLM applications shows how.

#FAQ

#Do LLMs understand language?

They build rich internal representations that capture meaning well enough to translate, summarize and reason about text. Whether that counts as "understanding" is partly a philosophical question. Practically, they are very capable pattern learners that still fail in ways humans wouldn't, so production systems need checks.

#What is the difference between parameters and tokens?

Parameters are the learned weights inside the model; they're fixed after training. Tokens are the units of text the model reads and writes at run time. A model's parameter count describes its size, while token counts describe how much text a request processes.

#Why do LLMs give different answers to the same prompt?

Because generation samples from a probability distribution. With temperature above zero, different tokens can be picked on each run. Even at temperature zero, small numerical differences in batched GPU inference can occasionally change outputs.

#Can an LLM learn from my conversations?

Not during a conversation: the weights don't change at inference time. The model only "remembers" what is inside its current context window. Whether a provider uses your data for future training depends on its policies and your account settings.