Transformers - SustainSys AI Academy

Introduction to Transformers

The Transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., has fundamentally reshaped artificial intelligence. It replaced recurrent neural networks (RNNs) and LSTMs as the dominant architecture for sequence processing and now powers virtually every major AI system — from GPT-4 and Claude to Stable Diffusion and AlphaFold.

Unlike RNNs that process sequences step-by-step (creating bottlenecks with long sequences), transformers process entire sequences in parallel using a mechanism called self-attention. This allows each token in the input to directly attend to every other token, capturing long-range dependencies without the vanishing gradient problem that plagued earlier architectures.

The key innovation is elegant: instead of sequential processing, transformers compute attention scores between all pairs of positions simultaneously, enabling massive parallelization on GPUs and making it feasible to train on enormous datasets. This architectural choice, combined with scale, is what enabled the current AI revolution.

What You'll Learn

Self-Attention Mechanism

Understand how queries, keys, and values enable tokens to attend to each other — the mathematical core of transformers.

Full Architecture

Learn the complete encoder-decoder structure including multi-head attention, feed-forward networks, layer normalization, and positional encoding.

Modern Innovations

Explore cutting-edge improvements: Flash Attention, Rotary Position Embeddings (RoPE), Grouped Query Attention (GQA), and Mixture of Experts.

Hands-On Code

Build transformers from scratch in PyTorch and use HuggingFace for practical NLP tasks with real, working Python code.

Prerequisites

Basic understanding of Python, neural networks (layers, activation functions, backpropagation), and linear algebra (matrix multiplication, dot products). Familiarity with PyTorch is helpful but not required.

Why Transformers Matter

Transformers aren't just another neural network architecture — they are the architecture that enabled the modern AI revolution. Before 2017, state-of-the-art NLP relied on LSTMs and attention mechanisms bolted onto recurrent models. After transformers, the field underwent a paradigm shift that continues to accelerate.

The Impact in Numbers

The performance improvement transformers brought to NLP tasks was dramatic. Here's how transformer-based models compare to previous approaches across standard benchmarks:

Transformer (BERT)

94.9%

LSTM + Attention

82.1%

Vanilla LSTM

72.3%

CNN-based (TextCNN)

67.5%

Bag of Words

52.0%

GLUE Benchmark composite scores — higher is better

Why Transformers Won

Parallelization

Unlike RNNs that process tokens sequentially, transformers process all tokens simultaneously. Training a sequence of 512 tokens takes the same time as processing 1 token — a massive speedup on GPUs.

Long-Range Dependencies

Self-attention connects every token to every other token directly. A word at position 1 can attend to position 500 without information passing through 499 intermediate steps, solving the vanishing gradient problem.

Scalability

Transformers scale remarkably well with data and compute. Scaling laws show predictable performance gains as models grow larger, enabling organizations to invest confidently in larger models.

Transfer Learning

Pre-trained transformer models (BERT, GPT) capture rich language understanding that transfers to downstream tasks. Fine-tuning a pre-trained model often outperforms training from scratch with 100x the data.

Key Insight: The transformer's success isn't just about self-attention — it's about the combination of self-attention, positional encoding, layer normalization, and residual connections. Each component is essential, and removing any one significantly degrades performance.

Historical Evolution

The transformer didn't appear in a vacuum — it was the culmination of decades of research in sequence modeling. Understanding this history helps you appreciate why each design decision was made and what problems it solves.

1986
Simple RNN
Sequential processing, vanishing gradients

1997
LSTM
Gates control information flow, mitigates vanishing gradient

2014
GRU
Simplified LSTM with fewer parameters

2014
Seq2Seq + Attention
Bahdanau attention enables selective focus

2017
Transformer
"Attention Is All You Need" — no recurrence!

2018-19
BERT & GPT
Pre-training + fine-tuning revolution

2020+
GPT-3, Claude, LLMs
Scale enables emergent abilities

The Key Breakthroughs

Bahdanau Attention (2014)

The idea of “attention” — letting the decoder look at different parts of the encoder output — was the crucial precursor to transformers. But Bahdanau attention was still bolted onto RNNs, maintaining sequential computation. The transformer's innovation was making attention the only mechanism.

Self-Attention vs Cross-Attention

Before transformers, attention was used between encoder and decoder (cross-attention). The transformer introduced self-attention, where a sequence attends to itself. This is what allows a sentence to understand relationships between its own words — e.g., resolving what 'it' refers to in 'The animal didn't cross the street because it was too tired.'

The transformer paper achieved state-of-the-art results on machine translation (English-German and English-French) while training significantly faster than the best RNN-based models. But the real impact came when researchers realized transformers could be pre-trained on massive text corpora and then fine-tuned for specific tasks — leading to BERT (2018) and GPT (2018-2019).

Core Concepts and Theory

The transformer architecture rests on a few powerful ideas. Let's build understanding from the ground up.

1. Self-Attention: The Key Innovation

Self-attention lets every position in a sequence compute a weighted sum of all positions. For each token, it asks: "How much should I pay attention to every other token in this sequence?"

This is computed using three learned projections of each input token:

Query (Q)

What am I looking for? The query vector represents what this token wants to know about other tokens. Think of it as a search query.

Key (K)

What do I contain? The key vector represents what information this token has to offer. Think of it as a search index entry.

Value (V)

What information do I pass along? Once attention scores are computed, the value vector provides the actual content that flows forward.

Scaled Dot-Product Attention

Attention(Q, K, V) = softmax(Q · K^T / √d_k) · V Where: Q = Query matrix (n × d_k) K = Key matrix (n × d_k) V = Value matrix (n × d_v) d_k = dimension of keys (scaling factor prevents softmax saturation)

The scaling factor √d_k is crucial. Without it, when the dimension d_k is large, the dot products between Q and K become very large, pushing the softmax function into regions with extremely small gradients, effectively killing the training signal.

Intuition: Imagine you're at a conference and need information. The Query is your question. Each person's Key is their name badge showing their expertise. The attention score tells you who to listen to most. The Value is what they actually tell you. You combine all responses weighted by how relevant each person is.

2. Multi-Head Attention

Instead of computing attention once, the transformer runs attention multiple times in parallel (called "heads"), each with different learned projections. This allows the model to attend to different types of relationships simultaneously.

Multi-Head Attention

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W_O Where each head_i = Attention(Q · W_Q_i, K · W_K_i, V · W_V_i) With h = number of heads (typically 8, 12, or 16) d_k = d_model / h (each head operates on a smaller dimension)

Why Multiple Heads?

Different heads learn to capture different types of relationships. In language, one head might capture syntactic dependencies (subject-verb agreement), another might capture coreference (what 'it' refers to), and another might capture semantic similarity. Having multiple heads gives the model richer representational capacity.

3. Positional Encoding

Since self-attention treats the input as a set (not a sequence), the model has no inherent notion of token order. Positional encoding injects position information into the input embeddings.

Sinusoidal Positional Encoding (Original)

PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) Where: pos = position in the sequence (0, 1, 2, ...) i = dimension index d_model = model embedding dimension

The sinusoidal encoding has an elegant property: for any fixed offset k, PE(pos+k) can be represented as a linear function of PE(pos), enabling the model to learn to attend to relative positions.

Modern Alternatives

Modern transformers often use Rotary Position Embeddings (RoPE) instead of sinusoidal encoding. RoPE encodes position by rotating the query and key vectors in 2D subspaces, directly encoding relative position information in the attention computation. This enables better length generalization — models can handle longer sequences than they were trained on.

4. Layer Normalization and Residual Connections

Every sub-layer (attention or feed-forward) in a transformer is wrapped with a residual connection and layer normalization:

Residual + Layer Norm

output = LayerNorm(x + SubLayer(x)) Pre-Norm variant (used in GPT/LLaMA): output = x + SubLayer(LayerNorm(x)) RMSNorm variant (simpler, used in modern LLMs): RMSNorm(x) = x / RMS(x) * γ where RMS(x) = √(mean(x²))

Residual connections allow gradients to flow directly through the network (solving the degradation problem in very deep models), while layer normalization stabilizes the activations, enabling faster and more stable training.

Architecture Deep Dive

The original transformer uses an encoder-decoder architecture. Modern variants use encoder-only (BERT), decoder-only (GPT, Claude, LLaMA), or encoder-decoder (T5, BART). Let's examine the full architecture.

Transformer Architecture Visualization

ENCODER

Input Embedding + PE

↓

Multi-Head Self-Attention

+ Add & Norm

↓

Feed-Forward Network

+ Add & Norm

× N layers

→

DECODER

Output Embedding + PE

↓

Masked Multi-Head Self-Attention

+ Add & Norm

↓

Multi-Head Cross-Attention

+ Add & Norm

↓

Feed-Forward Network

+ Add & Norm

× N layers

Encoder Block

Each encoder block has two sub-layers: (1) multi-head self-attention and (2) a position-wise feed-forward network. Each sub-layer has a residual connection and layer normalization around it.

Self-Attention in the Encoder

In the encoder, self-attention is bidirectional — each token can attend to all other tokens in the sequence. This is why BERT (encoder-only) can use context from both left and right of each token, making it powerful for understanding tasks like classification and named entity recognition.

Decoder Block

The decoder has three sub-layers: (1) masked multi-head self-attention (prevents attending to future tokens), (2) multi-head cross-attention (attends to encoder output), and (3) feed-forward network.

Why Masking Matters

During training, the decoder sees the entire target sequence at once. Without masking, it could “cheat” by looking at future tokens. The causal mask ensures position i can only attend to positions ≤ i, preserving the autoregressive property needed for generation. This is implemented by setting future positions to -infinity before softmax.

Modern Architectural Variants

Variant	Architecture	Examples	Best For
Encoder-Only	Bidirectional self-attention, no decoder	BERT, RoBERTa, DeBERTa	Classification, NER, sentence embeddings
Decoder-Only	Causal (masked) self-attention only	GPT-4, Claude, LLaMA, Gemini	Text generation, chat, reasoning
Encoder-Decoder	Full architecture as in original paper	T5, BART, mBART	Translation, summarization, seq2seq

Key Components Deep Dive

1. Token Embeddings

Input tokens are converted to dense vectors via a learned embedding matrix. For a vocabulary of size V and embedding dimension d_model, this is a V × d_model matrix. Each token ID looks up its corresponding row.

Python — Token Embedding Layer
import torch
import torch.nn as nn

class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size, d_model):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.d_model = d_model

    def forward(self, x):
        # Scale embeddings by sqrt(d_model) as in the original paper
        return self.embedding(x) * (self.d_model ** 0.5)

# Example: vocab of 32000 tokens, embedding dim 512
embed = TokenEmbedding(vocab_size=32000, d_model=512)
tokens = torch.tensor([[1, 42, 1337, 5, 0]])  # batch of 1 sentence
embedded = embed(tokens)  # Shape: [1, 5, 512]
print(f"Input shape: {tokens.shape}")
print(f"Embedded shape: {embedded.shape}")

2. Positional Encoding Implementation

Python — Sinusoidal Positional Encoding
import torch
import torch.nn as nn
import math

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000, dropout=0.1):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        # Create positional encoding matrix
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(
            torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)
        )

        pe[:, 0::2] = torch.sin(position * div_term)  # Even dimensions
        pe[:, 1::2] = torch.cos(position * div_term)  # Odd dimensions
        pe = pe.unsqueeze(0)  # Add batch dimension: [1, max_len, d_model]

        self.register_buffer('pe', pe)  # Not a parameter, but saved in state_dict

    def forward(self, x):
        # x shape: [batch_size, seq_len, d_model]
        x = x + self.pe[:, :x.size(1), :]
        return self.dropout(x)

# Example usage
pos_enc = PositionalEncoding(d_model=512, max_len=1000)
x = torch.randn(2, 100, 512)  # batch=2, seq_len=100, d_model=512
output = pos_enc(x)
print(f"Output shape: {output.shape}")  # [2, 100, 512]

3. Feed-Forward Network (FFN)

Each transformer layer includes a position-wise feed-forward network applied identically to each position. It typically expands the dimension by 4x, applies a non-linearity, then projects back down.

Feed-Forward Network

FFN(x) = max(0, x · W₁ + b₁) · W₂ + b₂ Dimensions: W₁: d_model × d_ff (expand: 512 → 2048) W₂: d_ff × d_model (contract: 2048 → 512) d_ff is typically 4 × d_model Modern variant (SwiGLU, used in LLaMA): FFN(x) = (Swish(x · W₁) ⊙ (x · W₃)) · W₂

4. Layer Normalization

Pre-Norm vs Post-Norm

The original paper uses Post-Norm: LayerNorm(x + SubLayer(x)). Modern LLMs like GPT and LLaMA use Pre-Norm: x + SubLayer(LayerNorm(x)). Pre-Norm is more stable during training and doesn't require learning rate warmup, but may produce slightly lower quality. Most practitioners now use Pre-Norm for its training stability.

5. The Output Layer

The final layer projects the decoder output (dimension d_model) back to vocabulary size using a linear layer followed by softmax to produce probability distributions over the next token. In many models, this output projection matrix shares weights with the input embedding matrix (weight tying), reducing parameters.

Implementation Guide

Let's build a transformer from scratch in PyTorch. We'll implement each component step by step, then assemble them into a complete model.

Step 1: Scaled Dot-Product Attention

Python — Scaled Dot-Product Attention from Scratch
import torch
import torch.nn as nn
import torch.nn.functional as F
import math

def scaled_dot_product_attention(query, key, value, mask=None):
    """
    Compute scaled dot-product attention.

    Args:
        query: [batch, heads, seq_len, d_k]
        key:   [batch, heads, seq_len, d_k]
        value: [batch, heads, seq_len, d_v]
        mask:  [batch, 1, 1, seq_len] or [batch, 1, seq_len, seq_len]

    Returns:
        output: [batch, heads, seq_len, d_v]
        attention_weights: [batch, heads, seq_len, seq_len]
    """
    d_k = query.size(-1)

    # Step 1: Compute attention scores
    # Q @ K^T => [batch, heads, seq_len, seq_len]
    scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)

    # Step 2: Apply mask (if provided)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))

    # Step 3: Softmax to get attention weights
    attention_weights = F.softmax(scores, dim=-1)

    # Step 4: Multiply by values
    output = torch.matmul(attention_weights, value)

    return output, attention_weights

# Test it
batch, heads, seq_len, d_k = 2, 8, 10, 64
Q = torch.randn(batch, heads, seq_len, d_k)
K = torch.randn(batch, heads, seq_len, d_k)
V = torch.randn(batch, heads, seq_len, d_k)

output, weights = scaled_dot_product_attention(Q, K, V)
print(f"Output shape: {output.shape}")     # [2, 8, 10, 64]
print(f"Weights shape: {weights.shape}")   # [2, 8, 10, 10]
print(f"Weights sum per query: {weights.sum(dim=-1)[0, 0]}")  # Should be all 1s

Step 2: Multi-Head Attention Module

Python — Multi-Head Attention from Scratch
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads, dropout=0.1):
        super().__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"

        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        # Linear projections for Q, K, V and output
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

        self.dropout = nn.Dropout(dropout)

    def forward(self, query, key, value, mask=None):
        batch_size = query.size(0)

        # 1. Linear projections
        Q = self.W_q(query)  # [batch, seq_len, d_model]
        K = self.W_k(key)
        V = self.W_v(value)

        # 2. Reshape to [batch, num_heads, seq_len, d_k]
        Q = Q.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        K = K.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        V = V.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)

        # 3. Compute attention
        attn_output, attn_weights = scaled_dot_product_attention(
            Q, K, V, mask
        )

        # 4. Reshape back: [batch, seq_len, d_model]
        attn_output = attn_output.transpose(1, 2).contiguous()
        attn_output = attn_output.view(batch_size, -1, self.d_model)

        # 5. Final linear projection
        output = self.W_o(attn_output)
        return output, attn_weights

# Test
mha = MultiHeadAttention(d_model=512, num_heads=8)
x = torch.randn(2, 20, 512)  # batch=2, seq=20, dim=512
output, weights = mha(x, x, x)  # Self-attention: Q=K=V=x
print(f"MHA output shape: {output.shape}")      # [2, 20, 512]
print(f"Attention weights: {weights.shape}")     # [2, 8, 20, 20]

Step 3: Complete Transformer Block

Python — Full Transformer Encoder Block
class TransformerBlock(nn.Module):
    """Single transformer encoder block with pre-norm."""
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()

        # Multi-head attention
        self.attention = MultiHeadAttention(d_model, num_heads, dropout)

        # Feed-forward network
        self.feed_forward = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),           # Modern transformers use GELU instead of ReLU
            nn.Dropout(dropout),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout),
        )

        # Layer normalization (pre-norm style)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # Pre-norm self-attention with residual
        normed = self.norm1(x)
        attn_output, _ = self.attention(normed, normed, normed, mask)
        x = x + self.dropout(attn_output)

        # Pre-norm feed-forward with residual
        normed = self.norm2(x)
        ff_output = self.feed_forward(normed)
        x = x + ff_output

        return x

# Build a 6-layer transformer encoder
class TransformerEncoder(nn.Module):
    def __init__(self, vocab_size, d_model=512, num_heads=8,
                 d_ff=2048, num_layers=6, max_len=512, dropout=0.1):
        super().__init__()
        self.embedding = TokenEmbedding(vocab_size, d_model)
        self.pos_encoding = PositionalEncoding(d_model, max_len, dropout)
        self.layers = nn.ModuleList([
            TransformerBlock(d_model, num_heads, d_ff, dropout)
            for _ in range(num_layers)
        ])
        self.norm = nn.LayerNorm(d_model)

    def forward(self, x, mask=None):
        x = self.embedding(x)
        x = self.pos_encoding(x)
        for layer in self.layers:
            x = layer(x, mask)
        return self.norm(x)

# Create model and test
model = TransformerEncoder(vocab_size=32000, d_model=512, num_layers=6)
tokens = torch.randint(0, 32000, (2, 128))  # batch=2, seq_len=128
output = model(tokens)
print(f"Model output: {output.shape}")  # [2, 128, 512]

# Count parameters
params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {params:,}")  # ~44M parameters

Notice the pattern: Every transformer implementation follows the same recipe — (1) embed tokens, (2) add position info, (3) run through N identical blocks of attention + FFN with residuals and normalization. Understanding this pattern lets you read any transformer codebase.

Advanced Techniques

Modern transformers have evolved significantly from the original 2017 design. Here are the key innovations that power today's state-of-the-art models.

Technique	Problem Solved	Used In	Key Benefit
Flash Attention	Memory bottleneck in attention	Almost all modern LLMs	2-4x speedup, O(N) memory instead of O(N²)
RoPE	Length generalization	LLaMA, Mistral, Qwen	Better relative position encoding, extrapolation
Grouped Query Attention (GQA)	KV cache memory usage	LLaMA 2/3, Mistral	70%+ KV cache reduction vs multi-head
SwiGLU Activation	Better FFN non-linearity	LLaMA, PaLM	Improved training efficiency and quality
RMSNorm	Simpler normalization	LLaMA, Gemma	Faster than LayerNorm, equivalent quality
Mixture of Experts (MoE)	Scaling without proportional compute	Mixtral, GPT-4 (rumored)	Much larger models with same inference cost

Flash Attention Explained

Standard attention computes the full N×N attention matrix, requiring O(N²) memory. Flash Attention uses tiling and kernel fusion to compute attention in blocks, reducing memory to O(N) while being 2-4x faster by avoiding slow GPU memory reads/writes.

Flash Attention Key Idea

Instead of materializing the full N×N attention matrix in GPU HBM (slow memory), Flash Attention computes attention in tiles that fit in SRAM (fast on-chip memory). It uses the online softmax trick to compute exact attention without ever storing the full matrix. This is a hardware-aware algorithm — same math, dramatically better performance.

Grouped Query Attention (GQA)

In standard multi-head attention (MHA), each head has its own Q, K, and V projections. GQA shares K and V across groups of heads while keeping separate Q projections. This reduces the KV cache size during inference without significant quality loss.

Attention Variants Comparison

Multi-Head (MHA)

8 Q heads + 8 K heads + 8 V heads

KV cache: 8 × d_k per layer

100% KV memory

Grouped Query (GQA)

8 Q heads + 2 K heads + 2 V heads

KV cache: 2 × d_k per layer

25% KV memory

Multi-Query (MQA)

8 Q heads + 1 K head + 1 V head

KV cache: 1 × d_k per layer

12.5% KV memory

Comparison with Alternatives

How do transformers compare to the architectures they replaced, and to newer alternatives?

Feature	RNN / LSTM	Transformer	State Space (Mamba)
Sequence Processing	Sequential (O(N) steps)	Parallel (O(1) depth)	Parallel (O(N log N))
Attention Complexity	N/A (implicit via hidden state)	O(N²) per layer	O(N) linear
Long-Range Dependencies	Poor (vanishing gradients)	Excellent (direct connections)	Good (selective state space)
Training Speed	Slow (sequential)	Fast (highly parallel)	Fast (hardware-efficient)
Inference Speed	O(1) per token	O(N) per token (KV cache)	O(1) per token
Memory (Inference)	O(1) — fixed hidden state	O(N) — growing KV cache	O(1) — fixed state

The Transformer Trade-off: Transformers trade O(N²) attention cost for vastly superior parallel training and better quality. For sequences up to ~128K tokens, this trade-off is overwhelmingly worth it. For extremely long sequences (1M+ tokens), hybrid approaches combining transformers with linear attention or state space models may be needed.

Performance Comparison

GPT-4 (Transformer)

96%

Claude 3.5 (Transformer)

94%

Mamba-2 (State Space)

78%

RWKV-6 (Linear RNN)

72%

Average score across common reasoning benchmarks (approximate)

Real-World Use Cases

Transformers have expanded far beyond their original NLP origins. Here are the major domains where they dominate.

Natural Language Processing

The original domain: machine translation, text summarization, question answering, sentiment analysis, named entity recognition. Models like BERT, GPT, and T5 set the standard.

Conversational AI

ChatGPT, Claude, Gemini — all built on transformers. Long-context understanding, instruction following, reasoning, and multi-turn dialogue are transformer strengths.

Computer Vision

Vision Transformers (ViT) treat image patches as tokens. DINO, DINOv2, and SAM use transformers for self-supervised image understanding, object detection, and segmentation.

Code Generation

GitHub Copilot, CodeLlama, StarCoder — transformers trained on code can write, debug, explain, and refactor code across dozens of programming languages.

Scientific AI

AlphaFold uses transformers to predict protein 3D structure from amino acid sequences. ESMFold, RoseTTAFold extend this to drug discovery and molecular design.

Multimodal AI

Models like GPT-4V, Gemini, and LLaVA combine vision and language transformers to understand images, charts, documents, and answer questions about visual content.

Enterprise Applications

Deploying transformers in production requires careful consideration of latency, cost, scaling, and reliability. Here's a typical enterprise transformer inference pipeline:

Production Transformer Pipeline

📥 Request
Router

🔤 Tokenizer

💾 KV Cache
Lookup

🧠 GPU
Inference

📊 Post-
Process

📤 Response
Stream

Key Enterprise Considerations

Quantization

Reduce model precision from FP32/FP16 to INT8 or INT4 for 2-4x inference speedup with minimal quality loss. Tools: bitsandbytes, GPTQ, AWQ.

KV Cache Optimization

During autoregressive generation, cache key-value pairs from previous tokens. Use PagedAttention (vLLM) for efficient memory management across batches.

Speculative Decoding

Use a small 'draft' model to generate candidate tokens, then verify them in parallel with the large model. 2-3x speedup for generation.

Continuous Batching

Instead of waiting for entire batches to complete, dynamically add/remove requests from the batch. Dramatically improves throughput and latency.

Common Mistakes

Building and fine-tuning transformers involves many subtle pitfalls. Here are the most common mistakes practitioners make.

⚠️ Mistake #1: Forgetting to Scale Attention. Computing Q·K^T without dividing by √d_k causes the softmax to saturate, producing near-one-hot attention weights. This kills gradient flow and makes learning impossible. Always use scaled dot-product attention.

⚠️ Mistake #2: Wrong Masking. Forgetting causal masks in decoder models (or applying them incorrectly) lets the model cheat by looking at future tokens during training. The model will train well but generate garbage at inference time.

⚠️ Mistake #3: Ignoring Learning Rate Warmup. Transformers are sensitive to the initial learning rate. Without warmup (gradually increasing lr for the first few thousand steps), training often diverges. Use a warmup schedule of 1-5% of total training steps.

⚠️ Mistake #4: Applying Softmax Twice. Some loss functions (like nn.CrossEntropyLoss) include the softmax internally. Applying softmax to your model output AND using CrossEntropyLoss means you're applying softmax twice, destroying your loss landscape.

Performance Pitfalls

Sequence Length vs Memory

Attention is O(N²) in memory. A model that handles 512 tokens fine may OOM at 2048 tokens (using 16x more attention memory). Always profile memory usage at your target sequence length, not a shorter test length.

Tokenizer Mismatch

Using a tokenizer that wasn't trained with your model produces garbage. GPT-2's tokenizer and BERT's tokenizer produce completely different token IDs for the same text. Always use the tokenizer that came with your model.

Not Using Mixed Precision

Training transformers in FP32 is 2-3x slower and uses 2x more memory than FP16/BF16 mixed precision. Always use torch.cuda.amp.autocast() or accelerate's mixed precision for training.

Best Practices

Use Pre-Norm Architecture

Place LayerNorm before (not after) the attention and FFN sub-layers. This makes training more stable and eliminates the need for careful learning rate warmup.

Learning Rate: Cosine Schedule

Use a cosine learning rate schedule with warmup. Peak LR of 1e-4 to 3e-4 for most models. Warmup for 1-5% of total steps, then cosine decay to 1/10th of peak.

Gradient Clipping

Clip gradients to a max norm of 1.0. Transformer training can produce gradient spikes (especially early in training) that cause divergence without clipping.

Weight Initialization

Use Xavier/Glorot initialization for most weights. Scale residual path weights by 1/√N where N is the number of layers. This prevents the residual stream from growing too large.

Flash Attention

Always use Flash Attention when available (flash_attn, xformers, or PyTorch's scaled_dot_product_attention). It's strictly better — same results, less memory, faster.

Gradient Accumulation

If your batch can't fit in memory, use gradient accumulation over multiple micro-batches. Effective batch size = micro_batch × gradient_accumulation_steps × num_gpus.

The Golden Rule of Transformer Training

Start with a known-good configuration from an existing model's paper (learning rate, batch size, warmup steps, etc.) and modify one thing at a time. Transformer training is sensitive to hyperparameters, and debugging divergent training runs is difficult. Replicate before you innovate.

Advanced Insights

Scaling Laws

One of the most important discoveries in modern AI is that transformer performance follows predictable scaling laws. The "Chinchilla" scaling laws (Hoffmann et al., 2022) showed that optimal performance comes from scaling both model size and training data together.

Scaling Laws: Loss vs Compute

N

Model Parameters

Loss ∝ N^-0.076

D

Training Tokens

Loss ∝ D^-0.095

C

Total Compute

Optimal: N ∝ C^0.5, D ∝ C^0.5

Chinchilla Optimal Scaling

For compute budget C FLOPs: Optimal parameters N ≈ 0.0554 × C^0.5 Optimal tokens D ≈ 0.0554 × C^0.5 Rule of thumb: train on ~20 tokens per parameter 7B model → ~140B tokens 70B model → ~1.4T tokens

Emergent Abilities

What Are Emergent Abilities?

As transformers scale beyond certain thresholds, they exhibit capabilities that are absent in smaller models — seemingly appearing suddenly rather than gradually. Examples include in-context learning (few-shot prompting), chain-of-thought reasoning, and the ability to follow complex multi-step instructions. Whether these are truly 'emergent' or just hard to detect at small scale is an active research debate.

Attention Pattern Analysis

Researchers have found that transformer attention heads specialize into distinct roles. Common patterns include:

Positional Heads

Attend to specific relative positions (e.g., always attend to the previous token). These implement a form of n-gram model within the transformer.

Syntactic Heads

Capture grammatical relationships like subject-verb agreement, even across long distances. These heads implement parsing-like behavior.

Induction Heads

Implement in-context learning by recognizing and continuing patterns seen earlier in the context. Two attention heads work together: one finds a previous occurrence, the other copies what followed.

Python Code Examples

Here are practical, ready-to-run code examples for working with transformers in production.

1. Using HuggingFace Transformers Pipeline

Python — HuggingFace Quick Start
from transformers import pipeline

# Sentiment Analysis
classifier = pipeline("sentiment-analysis")
result = classifier("Transformers have revolutionized NLP!")
print(result)
# [{'label': 'POSITIVE', 'score': 0.9998}]

# Text Generation
generator = pipeline("text-generation", model="gpt2")
output = generator(
    "The transformer architecture",
    max_length=50,
    num_return_sequences=1,
    temperature=0.7,
)
print(output[0]["generated_text"])

# Question Answering
qa = pipeline("question-answering")
result = qa(
    question="What is self-attention?",
    context="Self-attention allows each token to attend to all other tokens "
            "in the sequence, computing weighted sums based on relevance."
)
print(f"Answer: {result['answer']} (score: {result['score']:.4f})")

2. Loading and Using Pre-trained Models

Python — Working with AutoModel and AutoTokenizer
from transformers import AutoTokenizer, AutoModel
import torch

# Load model and tokenizer
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Tokenize input
text = "Transformers process all tokens in parallel using self-attention."
inputs = tokenizer(
    text,
    return_tensors="pt",     # Return PyTorch tensors
    padding=True,
    truncation=True,
    max_length=128,
)

print(f"Input IDs: {inputs['input_ids'].shape}")
print(f"Tokens: {tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])}")

# Get model output
with torch.no_grad():
    outputs = model(**inputs)

# outputs.last_hidden_state: [batch, seq_len, hidden_dim]
print(f"Output shape: {outputs.last_hidden_state.shape}")

# Get [CLS] token embedding (useful for classification)
cls_embedding = outputs.last_hidden_state[:, 0, :]
print(f"CLS embedding: {cls_embedding.shape}")  # [1, 768]

3. Fine-Tuning with HuggingFace Trainer

Python — Fine-Tuning BERT for Classification
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer
)
from datasets import load_dataset
import numpy as np

# Load dataset
dataset = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Tokenize
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
        max_length=256,
    )

tokenized = dataset.map(tokenize_function, batched=True)

# Load model for classification (2 classes: pos/neg)
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)

# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    learning_rate=2e-5,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=True,  # Mixed precision training
)

# Metrics
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    accuracy = (predictions == labels).mean()
    return {"accuracy": accuracy}

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"].select(range(5000)),
    eval_dataset=tokenized["test"].select(range(1000)),
    compute_metrics=compute_metrics,
)

trainer.train()
print(trainer.evaluate())

4. Attention Visualization

Python — Visualizing Attention Patterns
import torch
from transformers import AutoTokenizer, AutoModel
import matplotlib.pyplot as plt
import seaborn as sns

# Load model with attention output
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, output_attentions=True)

# Process text
text = "The cat sat on the mat because it was tired"
inputs = tokenizer(text, return_tensors="pt")
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

with torch.no_grad():
    outputs = model(**inputs)

# outputs.attentions: tuple of (batch, heads, seq_len, seq_len)
# One tensor per layer
attentions = outputs.attentions

# Visualize attention in layer 6, head 0
layer, head = 5, 0
attn_matrix = attentions[layer][0, head].numpy()

plt.figure(figsize=(10, 8))
sns.heatmap(
    attn_matrix,
    xticklabels=tokens,
    yticklabels=tokens,
    cmap="Blues",
    annot=False,
)
plt.title(f"Attention Pattern - Layer {layer+1}, Head {head+1}")
plt.xlabel("Key (attended to)")
plt.ylabel("Query (attending from)")
plt.tight_layout()
plt.savefig("attention_heatmap.png", dpi=150)
plt.show()

# Check what "it" attends to (shows coreference resolution)
it_idx = tokens.index("it")
top_attended = torch.topk(attentions[layer][0, head, it_idx], k=3)
print(f"'it' most attends to: {[tokens[i] for i in top_attended.indices]}")

5. Efficient Inference with vLLM

Python — Production Inference with vLLM
# vLLM provides high-throughput LLM serving
# Install: pip install vllm

from vllm import LLM, SamplingParams

# Load model with automatic tensor parallelism
llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf",
    tensor_parallel_size=1,     # Number of GPUs
    dtype="float16",
    max_model_len=4096,
    gpu_memory_utilization=0.9,
)

# Define sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=256,
    stop=["\n\n"],
)

# Batch inference (much faster than sequential)
prompts = [
    "Explain self-attention in one paragraph:",
    "What is the difference between BERT and GPT?",
    "How does positional encoding work?",
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated = output.outputs[0].text
    print(f"Prompt: {prompt[:50]}...")
    print(f"Response: {generated[:200]}...")
    print("---")

Try It Yourself

Put your transformer knowledge into practice with these hands-on exercises. Each builds on the concepts covered above.

Exercise 1: Build Scaled Dot-Product Attention

Implement the scaled dot-product attention function from scratch. Verify it produces attention weights that sum to 1 for each query, and test with a causal mask.

Python — Starter Code
import torch
import torch.nn.functional as F
import math

def my_attention(query, key, value, mask=None):
    """
    TODO: Implement scaled dot-product attention.

    Steps:
    1. Compute scores = Q @ K^T / sqrt(d_k)
    2. Apply mask (set masked positions to -inf)
    3. Apply softmax
    4. Multiply by V
    """
    d_k = query.size(-1)
    # YOUR CODE HERE
    pass

# Test your implementation
Q = torch.randn(1, 1, 4, 8)  # batch=1, head=1, seq=4, d_k=8
K = torch.randn(1, 1, 4, 8)
V = torch.randn(1, 1, 4, 8)

# Create causal mask
mask = torch.tril(torch.ones(4, 4)).unsqueeze(0).unsqueeze(0)

output, weights = my_attention(Q, K, V, mask)
assert weights.sum(dim=-1).allclose(torch.ones(1, 1, 4)), "Weights should sum to 1!"
print("Attention weights (should be lower-triangular):")
print(weights[0, 0])

Exercise 2: Visualize Positional Encodings

Generate positional encodings for 100 positions with d_model=64. Plot the encodings as a heatmap to see the sinusoidal patterns. What patterns do you notice for different dimensions?

Python — Starter Code
import torch
import math
import matplotlib.pyplot as plt

def create_positional_encoding(max_len, d_model):
    """TODO: Create sinusoidal positional encodings."""
    pe = torch.zeros(max_len, d_model)
    # YOUR CODE HERE: Fill pe using sin/cos formulas
    return pe

# Generate and visualize
pe = create_positional_encoding(100, 64)
plt.figure(figsize=(12, 6))
plt.imshow(pe.numpy(), cmap='RdBu', aspect='auto')
plt.xlabel('Embedding Dimension')
plt.ylabel('Position')
plt.title('Positional Encoding Patterns')
plt.colorbar()
plt.show()

Exercise 3: Compare Tokenizers

Load the tokenizers for GPT-2, BERT, and LLaMA. Tokenize the same sentence with each and compare: How many tokens does each produce? How do they handle unknown words? What are the vocabulary sizes?

Python — Starter Code
from transformers import AutoTokenizer

sentence = "The transformer architecture revolutionized NLP in 2017."

tokenizers = {
    "GPT-2": "gpt2",
    "BERT": "bert-base-uncased",
    "LLaMA": "meta-llama/Llama-2-7b-hf",
}

for name, model_id in tokenizers.items():
    try:
        tok = AutoTokenizer.from_pretrained(model_id)
        tokens = tok.tokenize(sentence)
        ids = tok.encode(sentence)
        print(f"\n{name}:")
        print(f"  Vocab size: {tok.vocab_size:,}")
        print(f"  Tokens ({len(tokens)}): {tokens}")
        print(f"  IDs: {ids}")
    except Exception as e:
        print(f"  {name}: Requires authentication - {e}")

Exercise 4: Fine-tune a Sentiment Classifier

Fine-tune a DistilBERT model on the IMDB dataset for sentiment classification. Train for 1 epoch, then evaluate accuracy on the test set. Target: >88% accuracy.

Python — Starter Code
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer,
)
from datasets import load_dataset
import numpy as np

# TODO: Complete this fine-tuning pipeline
# 1. Load the "imdb" dataset
# 2. Tokenize with distilbert-base-uncased
# 3. Create model with num_labels=2
# 4. Set training args (1 epoch, batch_size=32, lr=2e-5)
# 5. Train and evaluate

# YOUR CODE HERE

Interview Questions

These questions are commonly asked in ML engineering and AI research interviews. Click each question to reveal a detailed answer.

Q1: Explain the self-attention mechanism. Why is it called 'self' attention? ▼

Self-attention computes attention scores between all pairs of positions within the same sequence (hence 'self'). Each position generates a Query (what it's looking for), Key (what it contains), and Value (what it passes along). Attention scores are computed as the scaled dot product between queries and keys, normalized with softmax, then used to compute weighted sums of values.

It's called 'self' because the queries, keys, and values all come from the same sequence, unlike cross-attention where queries come from one sequence and keys/values from another (e.g., decoder attending to encoder output).

Q2: Why does the transformer scale the dot product by √d_k? ▼

When the dimension d_k is large, the dot products Q·K^T grow in magnitude (variance proportional to d_k), pushing the softmax into regions with extremely small gradients. Dividing by √d_k normalizes the variance of the dot products to approximately 1, keeping softmax in a region with useful gradients. Without scaling, the model would produce near-one-hot attention distributions early in training, making optimization very difficult.

Q3: What is the computational complexity of self-attention, and why is it a problem? ▼

Self-attention has O(N² · d) time and O(N²) space complexity, where N is the sequence length and d is the dimension. For a sequence of 128K tokens, the attention matrix has 128K × 128K = 16.4 billion entries per head per layer. This quadratic scaling limits the maximum context length and makes long-sequence processing expensive. Solutions include Flash Attention (same math, better memory access patterns), sparse attention (attend to a subset of positions), and linear attention variants.

Q4: Compare encoder-only, decoder-only, and encoder-decoder architectures. ▼

Encoder-only (BERT): Bidirectional attention, sees full context. Best for understanding tasks (classification, NER, similarity). Cannot generate text autoregressively.

Decoder-only (GPT, Claude): Causal attention (each token sees only previous tokens). Best for generation tasks. Can also do understanding tasks via prompting. Dominant architecture for LLMs.

Encoder-decoder (T5): Encoder processes input bidirectionally, decoder generates output autoregressively with cross-attention to encoder. Best for sequence-to-sequence tasks (translation, summarization). More parameters for same capability.

Q5: What is the KV cache, and why does it matter for inference? ▼

During autoregressive generation, each new token needs to attend to all previous tokens. Without caching, the model would recompute the Key and Value projections for all previous tokens at every step — O(N²) total computation for N tokens. The KV cache stores previously computed K and V tensors, so only the new token's K and V need to be computed at each step, reducing generation to O(N) total computation. The trade-off is memory: the KV cache grows linearly with sequence length and is often the memory bottleneck during inference.

Q6: Explain Flash Attention. How does it achieve speedup without approximation? ▼

Flash Attention is an IO-aware exact attention algorithm. Standard attention materializes the full N×N attention matrix in GPU HBM (high bandwidth memory), which is slow to read/write. Flash Attention tiles the computation into blocks that fit in SRAM (on-chip memory), which is 10-100x faster. It uses the online softmax algorithm to compute exact softmax without materializing the full attention matrix. The result is mathematically identical to standard attention but 2-4x faster and uses O(N) memory instead of O(N²).

Q7: What are Rotary Position Embeddings (RoPE)? ▼

RoPE encodes position information by rotating query and key vectors in 2D subspaces. For each pair of dimensions, it applies a rotation by an angle proportional to the position. The key property: the dot product between a rotated query at position m and a rotated key at position n depends only on (m-n), automatically encoding relative position. Benefits: (1) naturally encodes relative position in the attention computation, (2) better length generalization than absolute position embeddings, (3) decays attention with distance, similar to a learnable bias. Used by LLaMA, Mistral, and most modern LLMs.

Q8: How do Mixture of Experts (MoE) transformers work? ▼

MoE replaces the dense FFN layer with multiple 'expert' FFN layers and a learned routing network. For each token, the router selects the top-k experts (typically k=2), and only those experts process the token. This allows the model to have many more total parameters (larger capacity) while keeping per-token computation constant. Mixtral 8x7B has 47B total parameters but only activates ~13B per token, achieving quality close to models 3x its active size. Challenges include load balancing across experts, training stability, and communication overhead in distributed settings.

Frequently Asked Questions

What's the difference between attention and self-attention? ▼

Attention is the general mechanism where one sequence attends to another. Self-attention is the specific case where a sequence attends to itself. In the original transformer, the encoder uses self-attention, while the decoder uses both self-attention (attending to previous decoder outputs) and cross-attention (attending to encoder outputs).

How many parameters does a transformer have? ▼

For a transformer with L layers, d_model dimension, h heads, and vocabulary V: approximately 12 × L × d_model² parameters (for attention + FFN), plus V × d_model for embeddings. BERT-base (12 layers, 768 dim) has ~110M parameters. GPT-3 (96 layers, 12288 dim) has 175B parameters. LLaMA-3 70B has 70B parameters.

Can transformers handle sequences of any length? ▼

In theory, the self-attention mechanism can handle any length. In practice, memory is O(N²) and computation is O(N² × d), limiting context length. Modern solutions include: Flash Attention (reduces memory to O(N)), position encoding that generalizes (RoPE with NTK-aware scaling), and architectural innovations (ring attention for distributed long-context). Current state-of-the-art models support 128K-1M token contexts.

Why are transformers so much better than RNNs? ▼

Three main reasons: (1) Parallelization — transformers process all positions simultaneously during training, making them dramatically faster on GPUs. (2) Long-range dependencies — every token directly attends to every other token, while RNNs must pass information through a bottleneck hidden state. (3) Scalability — transformers scale predictably with compute and data, following power law scaling curves that enable reliable investment in larger models.

What is the 'attention is all you need' claim really saying? ▼

The original paper's title argues that you don't need recurrence (RNNs) or convolution for sequence modeling — attention alone is sufficient. The model eliminates recurrent connections entirely, using only self-attention layers plus simple feed-forward networks. This was a bold claim at the time, as RNNs were considered essential for sequence processing. The paper proved this claim on machine translation tasks, and subsequent work has confirmed it across virtually all sequence modeling domains.

How do transformers handle different languages? ▼

Through subword tokenization and multilingual training. Tokenizers like BPE and SentencePiece split text into common subword units, naturally handling any script. Models like mBERT and XLM-RoBERTa are trained on text from 100+ languages simultaneously, learning shared representations. Remarkably, they exhibit cross-lingual transfer — fine-tuning on English data often improves performance on other languages too, especially for related languages.

Are transformers being replaced by newer architectures? ▼

As of 2025, transformers remain dominant for most AI tasks. State Space Models (SSMs) like Mamba offer O(N) complexity and show promise for very long sequences, but haven't matched transformer quality on reasoning-heavy tasks. Hybrid architectures (combining transformers with SSMs) are an active research area. The most likely near-term evolution is efficient transformer variants (Flash Attention, GQA, MoE) rather than complete replacement.

What hardware do I need to train a transformer? ▼

It depends on scale. Fine-tuning BERT or small models: a single GPU (16GB+ VRAM). Fine-tuning 7B parameter models with LoRA: 1 GPU with 24GB (e.g., RTX 4090). Pre-training 7B models: 8-64 GPUs for days/weeks. Pre-training 70B+ models: hundreds to thousands of high-end GPUs (H100, A100) for months. Cloud providers (AWS, GCP, Lambda Labs) make GPU access affordable for smaller-scale work.

What's the difference between GPT and BERT? ▼

GPT is decoder-only with causal (left-to-right) attention — designed for generation. BERT is encoder-only with bidirectional attention — designed for understanding. GPT is pre-trained to predict the next token. BERT is pre-trained with masked language modeling (predicting masked tokens). In practice, GPT-style models have won because they can do both generation AND understanding via prompting, while BERT can only do understanding.

How do I choose which transformer model to use for my task? ▼

For text classification/NER: start with a BERT-family model (DeBERTa-v3 for best quality). For text generation/chatbots: use a decoder model (LLaMA 3, Mistral, or API-based GPT-4/Claude). For embeddings/similarity: use a sentence transformer (all-MiniLM-L6-v2 for speed, gte-large for quality). For translation: use an encoder-decoder (NLLB, mBART). Always start with the smallest model that meets your quality requirements — it's cheaper and faster to iterate.

Summary and Key Takeaways

Self-Attention Is the Core

Transformers use self-attention to let every token attend to every other token in parallel. The Q, K, V mechanism with scaled dot products is the mathematical foundation that enables this.

Architecture Matters

The combination of multi-head attention, feed-forward networks, residual connections, and layer normalization — not just attention alone — is what makes transformers work. Each component is essential.

Scale Is Key

Transformer performance scales predictably with model size and training data. The Chinchilla scaling laws guide efficient allocation of compute budget between parameters and training tokens.

Modern Innovations

Flash Attention, RoPE, GQA, and MoE have dramatically improved transformer efficiency. These innovations enable longer contexts, faster inference, and larger models without proportional compute increases.

Practical Skills

Using HuggingFace for fine-tuning, understanding tokenization, implementing attention from scratch, and choosing the right model for your task are the essential practical skills.

Resources and Further Reading

Essential Papers

Attention Is All You Need (2017)

Vaswani et al. — The foundational transformer paper. Start here to understand the original architecture.

BERT (2018)

Devlin et al. — Bidirectional encoder representations. Introduced pre-training + fine-tuning paradigm.

GPT-3 (2020)

Brown et al. — Demonstrated few-shot learning and emergent abilities at scale.

Flash Attention (2022)

Dao et al. — IO-aware exact attention algorithm. Essential reading for efficiency.

Chinchilla (2022)

Hoffmann et al. — Optimal scaling laws for compute-efficient training.

LLaMA (2023)

Touvron et al. — Open-weight transformer with modern architecture choices. Good reference implementation.