Map
Written pages are links. The other titles are not written yet.
Beginners
- How a GPT works
A GPT-style model repeats one step: it turns the current tokens into a distribution over the next token, appends one token, and repeats.
- Using a chatbot
A chatbot sends the conversation so far to a model and appends the model’s new tokens to that conversation.
- Prompts
A prompt is the token sequence already present when generation starts.
- Tokens
A token is an item in a fixed vocabulary, and a piece of text is a sequence of those items’ ids.
- The context window
The context window is the maximum number of tokens the model will take at one step.
- Likelihood and truth
The training target is the next token in the text, so a fluent continuation is not a check that the continuation matches the world.
- What a chat request contains
A chat request contains the text the product packs for the model, and a hosted product processes that text on the operator’s computers.
- A model is a function
During a reply, the model is a fixed function from the current token ids to a distribution over the next id.
- Training and inference
Training changes parameters so the next-token distributions match the training text. Inference uses the parameters and does not change them.
- Vectors
A vector is an ordered list of numbers. The dot product multiplies matching positions and adds those products.
- Text becomes numbers
Each token id selects a vector from an embedding table, and GPT-2 adds a second vector that depends on the position.
- The next token
The model produces one score per vocabulary id, and softmax turns those scores into probabilities that sum to 1.
- Attention
Attention builds a new vector for each token as a weighted sum of value vectors from tokens it is allowed to read.
- The transformer
A GPT-2 transformer is a stack of blocks. Each block mixes tokens with attention and then passes each position through the same feed-forward network.
- Temperature
Temperature divides the logits by a positive number before softmax. Above 1 the distribution flattens. Below 1 it sharpens.
- Parameters
A parameter is a number stored in the model. The forward pass reads the set, and training writes it.
- GPT-2 and later models
GPT-2 small is a public transformer this site can open. Later models keep that stack and change named pieces. A chat product adds training and serving around a model.
Technical
Numbers
- Vector
A vector is a one-dimensional array of real numbers with a fixed length.
Not written yet
- Matrix
A matrix is a rectangular array. An m×n matrix times an n-vector produces an m-vector.
Not written yet
- Dot product
The dot product of two equal-length vectors is the sum of the products of matching entries.
Not written yet
- Shape
A shape is the list of axis lengths of a tensor.
Not written yet
- Probability
A discrete distribution on a finite set assigns non-negative numbers that sum to 1.
Not written yet
- Softmax
Softmax maps real scores to positive numbers that sum to 1.
Not written yet
- Logarithm
The natural log turns a product of probabilities into a sum, which is how the next-token loss is written.
Not written yet
- Gradient
The gradient is the direction of steepest increase. A training step moves the other way.
Not written yet
Data
- Token
A GPT-2 token is an integer id from 0 through 50256 from the model’s byte-level BPE tokenizer.
Not written yet
- Byte-pair encoding
BPE repeatedly merges the most frequent adjacent pair, starting from bytes or characters.
Not written yet
- Vocabulary
The vocabulary is the ordered set that indexes embedding rows and the logit axis.
Not written yet
- Embedding
An embedding maps an id to a row of a learned matrix.
Not written yet
- Positional encoding
GPT-2 adds a learned vector for the position index, from 0 to 1023.
Not written yet
Network
- Linear layer
A linear layer computes Wx + b.
Not written yet
- GELU
GPT-2 applies the tanh approximation of GELU between the two feed-forward linear layers.
Not written yet
- Residual
A residual connection sets y = x + f(x).
Not written yet
- Layer norm
GPT-2 layer norm recenters a 768-vector, scales by its spread, then applies a learned weight and bias.
Not written yet
- Dropout
Dropout zeros a random fraction of values during training. At inference it is the identity.
Not written yet
Attention
- Query, key, and value
Attention projects each hidden vector into a query, a key, and a value.
Not written yet
- Attention scores
A score is a query-key dot product divided by √d, with d = 64 in GPT-2 small.
Not written yet
- Attention head
One head turns scaled scores into weights and outputs a weighted sum of values.
Not written yet
- Multi-head attention
GPT-2 runs 12 heads on slices of the 768-vector and concatenates their outputs back to 768.
Not written yet
- Causal mask
The causal mask stops a position from reading later positions by setting those scores to −∞ before softmax.
Not written yet
Block
- Feed-forward network
The GPT-2 feed-forward network is Linear 768→3072, GELU, Linear 3072→768, applied independently at each position.
Not written yet
- Transformer block
One GPT-2 block is attention and a feed-forward network, each preceded by layer norm and wrapped in a residual add.
Not written yet
- Stack
GPT-2 small runs 12 blocks, then a final layer norm, then the tied embedding to produce logits.
Not written yet
GPT-2
- GPT-2 config
GPT-2 small is the configuration n_layer 12, n_head 12, n_embd 768, n_ctx 1024, vocab 50257.
Not written yet
- Weight names
Each GPT-2 parameter tensor has a name and a shape in the checkpoint.
Not written yet
- Forward pass
The forward pass maps token ids to logits with the stack on this site’s pages.
Not written yet
- Logits
Logits are the 50257 real scores at a position, before softmax.
Not written yet
- Cross-entropy
The next-token loss is the negative log of the softmax probability of the id that actually came next.
Not written yet
- Training step
One training step runs the forward pass, the cross-entropy, a backward pass, and a parameter update.
Not written yet
- Sampling
Sampling turns the next-token distribution into one id: greedy, temperature, top-k, or top-p.
Not written yet
- KV cache
A KV cache stores past keys and values so the next token does not recompute them.
Not written yet
Training
- Dataset
GPT-2 is trained to predict the next token in documents. The paper’s corpus is called WebText.
Not written yet
- Batch
A batch is several token sequences stacked on a batch axis and run through the same forward pass.
Not written yet
- Learning rate
The learning rate scales the parameter step. GPT-2’s paper describes a warmup and a decay. Quote that schedule from the paper if you state numbers.
Not written yet
- Overfitting
Overfitting is a falling training loss while loss on held-out text stops falling or rises.
Not written yet
- Checkpoint
A checkpoint is a saved copy of the parameters, and usually the step count.
Not written yet
Modern
- Rotary position
Rotary position embeddings rotate the query and key at each position so their dot product depends on the relative offset.
Not written yet
- RMSNorm
RMSNorm divides a vector by its root-mean-square and multiplies by a learned scale. It does not subtract the mean.
Not written yet
- SwiGLU
A SwiGLU feed-forward multiplies a gated linear path by a SiLU path, using three matrices instead of GPT-2’s two.
Not written yet
- Grouped-query attention
Grouped-query attention uses fewer key and value heads than query heads. Several query heads share one key/value head.
Not written yet
- Mixture of experts
A mixture-of-experts layer picks a few feed-forward networks per token with a router, instead of running one feed-forward network.
Not written yet
- Context length
Self-attention’s score matrix grows with the square of the sequence length. GPT-2’s length is 1024. Later models use longer limits.
Not written yet
- Quantization
Quantization stores weights as small integers plus a scale, so the loaded numbers are an approximation of the original floats.
Not written yet
- Inference batching
Inference batching runs several requests through the same matrix multiplies. Continuous batching adds a new request when another request finishes a token.
Not written yet
- Supervised fine-tuning
Supervised fine-tuning continues next-token training on pairs of a prompt and a desired answer.
Not written yet
- Reinforcement learning from human feedback
RLHF trains a reward model on human comparisons, then updates the language model so that higher-reward answers become more likely.
Not written yet
- Direct preference optimization
DPO updates the language model on preferred and rejected answers directly, without a separate reward model at training time.
Not written yet
- Tool use
Tool use is a loop: the model emits a call, a program returns text, and that text is appended before the model continues.
Not written yet
- Reasoning models
A reasoning model spends extra generated tokens on intermediate steps before the tokens it presents as the answer.
Not written yet
Tools
- Vector dot product
The tool multiplies two equal-length vectors position by position and adds the products.
- Softmax
The tool turns a list of scores into probabilities that sum to 1.
- Tokenizer
The tool encodes text with the GPT-2 tokenizer and shows the token strings and ids.
- Temperature
The tool divides logits by a temperature and shows the softmax.
- Matrix multiply
The tool multiplies a small matrix by a vector and shows each row’s dot product.
Not written yet
- Attention heatmap
The tool shows a causal attention weight matrix for a typed list of scores.
Not written yet
- BPE trainer
The tool runs byte-pair encoding on a fixed toy corpus and lists each merge.
Not written yet
- Positional encoding
The tool adds a token vector and a position vector elementwise.
Not written yet
- Nearest embedding
The tool lists the nearest tokens to a chosen token in a committed slice of embeddings.
Not written yet
- Forward-pass stepper
The tool steps through a tiny forward pass with hidden size 4 and a 4-word vocabulary.
Not written yet
- Sampling compare
The tool shows greedy, temperature, top-k, and top-p on one list of logits.
Not written yet
Explorer
- Explorer
The explorer shows one precomputed forward pass of GPT-2 small on a fixed sentence.
Not written yet
- Model
GPT-2 small’s configuration, license lines copied from its model card, and where the trace came from.
Not written yet
- Trace
The fixed sentence split into GPT-2 tokens, with the ids from the trace file.
Not written yet
- Layers
For each of the 12 blocks, the size of the residual stream at each token in the trace.
Not written yet
- Attention
Attention weights for every head on the fixed sentence.
Not written yet
- Weights
Every parameter name and shape, plus a short preview taken from the checkpoint.
Not written yet