learn me a gpt

Map

Written pages are links. The other titles are not written yet.

Beginners

  • How a GPT works

    A GPT-style model repeats one step: it turns the current tokens into a distribution over the next token, appends one token, and repeats.

  • Using a chatbot

    A chatbot sends the conversation so far to a model and appends the model’s new tokens to that conversation.

  • Prompts

    A prompt is the token sequence already present when generation starts.

  • Tokens

    A token is an item in a fixed vocabulary, and a piece of text is a sequence of those items’ ids.

  • The context window

    The context window is the maximum number of tokens the model will take at one step.

  • Likelihood and truth

    The training target is the next token in the text, so a fluent continuation is not a check that the continuation matches the world.

  • What a chat request contains

    A chat request contains the text the product packs for the model, and a hosted product processes that text on the operator’s computers.

  • A model is a function

    During a reply, the model is a fixed function from the current token ids to a distribution over the next id.

  • Training and inference

    Training changes parameters so the next-token distributions match the training text. Inference uses the parameters and does not change them.

  • Vectors

    A vector is an ordered list of numbers. The dot product multiplies matching positions and adds those products.

  • Text becomes numbers

    Each token id selects a vector from an embedding table, and GPT-2 adds a second vector that depends on the position.

  • The next token

    The model produces one score per vocabulary id, and softmax turns those scores into probabilities that sum to 1.

  • Attention

    Attention builds a new vector for each token as a weighted sum of value vectors from tokens it is allowed to read.

  • The transformer

    A GPT-2 transformer is a stack of blocks. Each block mixes tokens with attention and then passes each position through the same feed-forward network.

  • Temperature

    Temperature divides the logits by a positive number before softmax. Above 1 the distribution flattens. Below 1 it sharpens.

  • Parameters

    A parameter is a number stored in the model. The forward pass reads the set, and training writes it.

  • GPT-2 and later models

    GPT-2 small is a public transformer this site can open. Later models keep that stack and change named pieces. A chat product adds training and serving around a model.

Technical

Numbers

  • Vector

    A vector is a one-dimensional array of real numbers with a fixed length.

    Not written yet

  • Matrix

    A matrix is a rectangular array. An m×n matrix times an n-vector produces an m-vector.

    Not written yet

  • Dot product

    The dot product of two equal-length vectors is the sum of the products of matching entries.

    Not written yet

  • Shape

    A shape is the list of axis lengths of a tensor.

    Not written yet

  • Probability

    A discrete distribution on a finite set assigns non-negative numbers that sum to 1.

    Not written yet

  • Softmax

    Softmax maps real scores to positive numbers that sum to 1.

    Not written yet

  • Logarithm

    The natural log turns a product of probabilities into a sum, which is how the next-token loss is written.

    Not written yet

  • Gradient

    The gradient is the direction of steepest increase. A training step moves the other way.

    Not written yet

Data

  • Token

    A GPT-2 token is an integer id from 0 through 50256 from the model’s byte-level BPE tokenizer.

    Not written yet

  • Byte-pair encoding

    BPE repeatedly merges the most frequent adjacent pair, starting from bytes or characters.

    Not written yet

  • Vocabulary

    The vocabulary is the ordered set that indexes embedding rows and the logit axis.

    Not written yet

  • Embedding

    An embedding maps an id to a row of a learned matrix.

    Not written yet

  • Positional encoding

    GPT-2 adds a learned vector for the position index, from 0 to 1023.

    Not written yet

Network

  • Linear layer

    A linear layer computes Wx + b.

    Not written yet

  • GELU

    GPT-2 applies the tanh approximation of GELU between the two feed-forward linear layers.

    Not written yet

  • Residual

    A residual connection sets y = x + f(x).

    Not written yet

  • Layer norm

    GPT-2 layer norm recenters a 768-vector, scales by its spread, then applies a learned weight and bias.

    Not written yet

  • Dropout

    Dropout zeros a random fraction of values during training. At inference it is the identity.

    Not written yet

Attention

  • Query, key, and value

    Attention projects each hidden vector into a query, a key, and a value.

    Not written yet

  • Attention scores

    A score is a query-key dot product divided by √d, with d = 64 in GPT-2 small.

    Not written yet

  • Attention head

    One head turns scaled scores into weights and outputs a weighted sum of values.

    Not written yet

  • Multi-head attention

    GPT-2 runs 12 heads on slices of the 768-vector and concatenates their outputs back to 768.

    Not written yet

  • Causal mask

    The causal mask stops a position from reading later positions by setting those scores to −∞ before softmax.

    Not written yet

Block

  • Feed-forward network

    The GPT-2 feed-forward network is Linear 768→3072, GELU, Linear 3072→768, applied independently at each position.

    Not written yet

  • Transformer block

    One GPT-2 block is attention and a feed-forward network, each preceded by layer norm and wrapped in a residual add.

    Not written yet

  • Stack

    GPT-2 small runs 12 blocks, then a final layer norm, then the tied embedding to produce logits.

    Not written yet

GPT-2

  • GPT-2 config

    GPT-2 small is the configuration n_layer 12, n_head 12, n_embd 768, n_ctx 1024, vocab 50257.

    Not written yet

  • Weight names

    Each GPT-2 parameter tensor has a name and a shape in the checkpoint.

    Not written yet

  • Forward pass

    The forward pass maps token ids to logits with the stack on this site’s pages.

    Not written yet

  • Logits

    Logits are the 50257 real scores at a position, before softmax.

    Not written yet

  • Cross-entropy

    The next-token loss is the negative log of the softmax probability of the id that actually came next.

    Not written yet

  • Training step

    One training step runs the forward pass, the cross-entropy, a backward pass, and a parameter update.

    Not written yet

  • Sampling

    Sampling turns the next-token distribution into one id: greedy, temperature, top-k, or top-p.

    Not written yet

  • KV cache

    A KV cache stores past keys and values so the next token does not recompute them.

    Not written yet

Training

  • Dataset

    GPT-2 is trained to predict the next token in documents. The paper’s corpus is called WebText.

    Not written yet

  • Batch

    A batch is several token sequences stacked on a batch axis and run through the same forward pass.

    Not written yet

  • Learning rate

    The learning rate scales the parameter step. GPT-2’s paper describes a warmup and a decay. Quote that schedule from the paper if you state numbers.

    Not written yet

  • Overfitting

    Overfitting is a falling training loss while loss on held-out text stops falling or rises.

    Not written yet

  • Checkpoint

    A checkpoint is a saved copy of the parameters, and usually the step count.

    Not written yet

Modern

  • Rotary position

    Rotary position embeddings rotate the query and key at each position so their dot product depends on the relative offset.

    Not written yet

  • RMSNorm

    RMSNorm divides a vector by its root-mean-square and multiplies by a learned scale. It does not subtract the mean.

    Not written yet

  • SwiGLU

    A SwiGLU feed-forward multiplies a gated linear path by a SiLU path, using three matrices instead of GPT-2’s two.

    Not written yet

  • Grouped-query attention

    Grouped-query attention uses fewer key and value heads than query heads. Several query heads share one key/value head.

    Not written yet

  • Mixture of experts

    A mixture-of-experts layer picks a few feed-forward networks per token with a router, instead of running one feed-forward network.

    Not written yet

  • Context length

    Self-attention’s score matrix grows with the square of the sequence length. GPT-2’s length is 1024. Later models use longer limits.

    Not written yet

  • Quantization

    Quantization stores weights as small integers plus a scale, so the loaded numbers are an approximation of the original floats.

    Not written yet

  • Inference batching

    Inference batching runs several requests through the same matrix multiplies. Continuous batching adds a new request when another request finishes a token.

    Not written yet

  • Supervised fine-tuning

    Supervised fine-tuning continues next-token training on pairs of a prompt and a desired answer.

    Not written yet

  • Reinforcement learning from human feedback

    RLHF trains a reward model on human comparisons, then updates the language model so that higher-reward answers become more likely.

    Not written yet

  • Direct preference optimization

    DPO updates the language model on preferred and rejected answers directly, without a separate reward model at training time.

    Not written yet

  • Tool use

    Tool use is a loop: the model emits a call, a program returns text, and that text is appended before the model continues.

    Not written yet

  • Reasoning models

    A reasoning model spends extra generated tokens on intermediate steps before the tokens it presents as the answer.

    Not written yet

Tools

  • Vector dot product

    The tool multiplies two equal-length vectors position by position and adds the products.

  • Softmax

    The tool turns a list of scores into probabilities that sum to 1.

  • Tokenizer

    The tool encodes text with the GPT-2 tokenizer and shows the token strings and ids.

  • Temperature

    The tool divides logits by a temperature and shows the softmax.

  • Matrix multiply

    The tool multiplies a small matrix by a vector and shows each row’s dot product.

    Not written yet

  • Attention heatmap

    The tool shows a causal attention weight matrix for a typed list of scores.

    Not written yet

  • BPE trainer

    The tool runs byte-pair encoding on a fixed toy corpus and lists each merge.

    Not written yet

  • Positional encoding

    The tool adds a token vector and a position vector elementwise.

    Not written yet

  • Nearest embedding

    The tool lists the nearest tokens to a chosen token in a committed slice of embeddings.

    Not written yet

  • Forward-pass stepper

    The tool steps through a tiny forward pass with hidden size 4 and a 4-word vocabulary.

    Not written yet

  • Sampling compare

    The tool shows greedy, temperature, top-k, and top-p on one list of logits.

    Not written yet

Explorer

  • Explorer

    The explorer shows one precomputed forward pass of GPT-2 small on a fixed sentence.

    Not written yet

  • Model

    GPT-2 small’s configuration, license lines copied from its model card, and where the trace came from.

    Not written yet

  • Trace

    The fixed sentence split into GPT-2 tokens, with the ids from the trace file.

    Not written yet

  • Layers

    For each of the 12 blocks, the size of the residual stream at each token in the trace.

    Not written yet

  • Attention

    Attention weights for every head on the fixed sentence.

    Not written yet

  • Weights

    Every parameter name and shape, plus a short preview taken from the checkpoint.

    Not written yet