Transformers have fundamentally changed how we approach sequence modeling, becoming the foundation of virtually all modern NLP systems. Introduced in the seminal paper “Attention Is All You Need” (2017), this architecture solved key limitations of RNNs while enabling unprecedented scale.
Why Transformers Matter
Before transformers, sequence models relied on recurrent connections (RNNs, LSTMs) that processed data sequentially. This created two major problems:
- No parallelization: Each timestep depends on the previous, limiting GPU utilization
- Long-range dependencies: Information from early timesteps gets diluted as it propagates through the sequence
Transformers solve both by using attention mechanisms that can directly relate any position to any other position, regardless of distance.
The Attention Mechanism
At the heart of transformers is scaled dot-product attention:
import torch
import torch.nn as nn
import math
def scaled_dot_product_attention(Q, K, V, mask=None):
"""
Q: Queries (batch_size, seq_len, d_k)
K: Keys (batch_size, seq_len, d_k)
V: Values (batch_size, seq_len, d_v)
"""
d_k = Q.size(-1)
# Compute attention scores
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
# Apply mask (for causal attention in decoders)
if mask is not None:
scores = scores.masked_fill(mask == 0, float('-inf'))
# Softmax to get attention weights
attention_weights = torch.softmax(scores, dim=-1)
# Weighted sum of values
output = torch.matmul(attention_weights, V)
return output, attention_weightsThe scaling factor sqrt(d_k) prevents dot products from growing too large in high dimensions, which would push softmax into regions with small gradients.
Multi-Head Attention
Rather than performing attention once, transformers use multiple attention heads that learn different types of relationships:
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, num_heads):
super().__init__()
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, query, key, value, mask=None):
batch_size = query.size(0)
# Linear projections and reshape for multi-head
Q = self.W_q(query).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
K = self.W_k(key).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
V = self.W_v(value).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
# Apply attention
attention_output, weights = scaled_dot_product_attention(Q, K, V, mask)
# Concatenate heads and final linear
attention_output = attention_output.transpose(1, 2).contiguous().view(
batch_size, -1, self.d_model
)
return self.W_o(attention_output)Multiple heads allow the model to jointly attend to information from different representation subspaces—one head might track syntactic relationships while another focuses on semantic similarity.
Positional Encoding
Unlike RNNs, transformers have no inherent sense of sequence order. Positional encodings inject position information into the input embeddings:
class PositionalEncoding(nn.Module):
def __init__(self, d_model, max_seq_length=5000):
super().__init__()
pe = torch.zeros(max_seq_length, d_model)
position = torch.arange(0, max_seq_length, dtype=torch.float).unsqueeze(1)
div_term = torch.exp(
torch.arange(0, d_model, 2).float() *
(-math.log(10000.0) / d_model)
)
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
self.register_buffer('pe', pe.unsqueeze(0))
def forward(self, x):
return x + self.pe[:, :x.size(1)]The sinusoidal functions allow the model to learn relative positions—PE(pos + k) can be represented as a linear function of PE(pos).
The Complete Transformer Block
A transformer layer combines attention with feed-forward networks and residual connections:
class TransformerBlock(nn.Module):
def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
super().__init__()
self.attention = MultiHeadAttention(d_model, num_heads)
self.feed_forward = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.ReLU(),
nn.Linear(d_ff, d_model)
)
self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)
self.dropout = nn.Dropout(dropout)
def forward(self, x, mask=None):
# Self-attention with residual
attn_output = self.attention(x, x, x, mask)
x = self.norm1(x + self.dropout(attn_output))
# Feed-forward with residual
ff_output = self.feed_forward(x)
x = self.norm2(x + self.dropout(ff_output))
return xKey design choices:
- LayerNorm before sublayers (Pre-LN): More stable training for deep models
- Residual connections: Enable gradient flow in deep networks
- Dropout: Regularization to prevent overfitting
Encoder-Decoder Architecture
The original transformer uses separate encoder and decoder stacks:
Encoder: Processes input sequence bidirectionally, creating contextualized representations
Decoder: Generates output autoregressively with:
- Masked self-attention (can’t look at future tokens)
- Cross-attention to encoder outputs
Modern variants simplify this:
- BERT: Encoder-only, trained with masked language modeling
- GPT: Decoder-only, trained with causal language modeling
- T5: Full encoder-decoder for sequence-to-sequence tasks
Applications Beyond NLP
Transformers have expanded far beyond text:
Vision Transformers (ViT): Treat image patches as sequences, achieving state-of-the-art results on ImageNet
Multimodal Models: CLIP, DALL-E, and GPT-4V combine vision and language
Protein Folding: AlphaFold uses transformer variants for predicting 3D structures
Time Series: Informer and Autoformer adapt attention for temporal data
Training Considerations
Warmup Schedule: Transformers require learning rate warmup to stabilize early training:
# Linear warmup then decay
scheduler = torch.optim.lr_scheduler.LambdaLR(
optimizer,
lr_lambda=lambda step: min(step / warmup_steps, 1.0) *
(0.5 * (1 + math.cos(math.pi * step / total_steps)))
)Gradient Accumulation: Essential for training with limited GPU memory
Mixed Precision: FP16 training is standard for large transformers
Scaling Laws
Transformers exhibit predictable scaling relationships:
- Loss ∝ Model Size^(-0.076): Larger models generalize better
- Loss ∝ Dataset Size^(-0.095): More data continues to help
- Loss ∝ Compute^(-0.050): Returns diminish but remain positive
These relationships have driven the development of models with hundreds of billions of parameters.
Modern Variants
Efficient Attention:
- FlashAttention: Memory-efficient exact attention
- Linear Attention: Approximate attention in O(n) time
- Sparse Attention: Only attend to local windows or strided patterns
Parameter Efficiency:
- LoRA: Low-rank adaptation for fine-tuning
- Prefix Tuning: Learnable prompt embeddings
- QLoRA: Quantized LoRA for consumer GPU fine-tuning
Conclusion
Transformers represent a paradigm shift in machine learning—from sequential processing to parallelizable attention. Their flexibility and scalability have made them the dominant architecture across modalities.
Understanding transformers is essential for modern ML practitioners. The concepts—attention, positional encoding, multi-head processing—appear in virtually every state-of-the-art system.
For those building with transformers today, frameworks like Hugging Face Transformers provide pre-trained models and easy fine-tuning APIs, while libraries like DeepSpeed and Megatron-LM enable training at massive scale.