PyTorch from Scratch
A Complete Beginner's Guide

What is PyTorch?

PyTorch is a Python library for building AI systems. Specifically, it makes it easy to build and train neural networks — mathematical models that learn patterns from data.

Think of it this way: teaching a computer to recognise cats means showing it thousands of photos, letting it make guesses, telling it how wrong it was, and letting it adjust. PyTorch is the toolkit that makes this entire process efficient and manageable.

Four Things People Confuse

Where PyTorch Fits in the AI Workflow

Real-World Applications

Tensors — The Core Data Structure

Everything in PyTorch is built on tensors. A tensor is like a Python list — but supercharged. It can hold millions of numbers, do maths on all of them simultaneously, and run on a GPU.

  Scalar (0D):    7

  Vector (1D):    [1, 2, 3]

  Matrix (2D):    [[1, 2, 3],
                   [4, 5, 6]]

  3D Tensor:      [[[1, 2], [3, 4]],
                    [[5, 6], [7, 8]]]

  4D Tensor:      Used for batches of images → (batch, channels, height, width)
      

Why Tensors Instead of Lists?

Creating Tensors

import torch

# From data you provide
scalar = torch.tensor(7)
vector = torch.tensor([1.0, 2.0, 3.0])
matrix = torch.tensor([[1.0, 2.0], [3.0, 4.0]])

# Useful creation functions
zeros  = torch.zeros(3, 4)       # all zeros, shape (3,4)
ones   = torch.ones(2, 3)        # all ones, shape (2,3)
rng    = torch.arange(0, 10, 2) # [0, 2, 4, 6, 8]
rand   = torch.rand(3, 3)       # random 0-1, uniform
randn  = torch.randn(3, 3)      # random, normal distribution

# Inspect a tensor
print(matrix.shape)   # torch.Size([2, 2])
print(matrix.ndim)    # 2
print(matrix.dtype)   # torch.float32

Data Types — Why dtype Matters

Neural networks almost always use float32 (decimal numbers). Feeding integer tensors where float32 is expected is one of the most common beginner errors.

Tensor Operations, Reshaping & Indexing

Basic Arithmetic

Operations happen element-wise — PyTorch applies the operation to every number simultaneously, with no loops needed.

import torch

a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([4.0, 5.0, 6.0])

print(a + b)   # tensor([5., 7., 9.])
print(a * b)   # tensor([ 4., 10., 18.]) — element-wise
print(a ** 2)  # tensor([1., 4., 9.])
print(a * 10)  # tensor([10., 20., 30.]) — broadcasting

Matrix Multiplication — The Engine of Deep Learning

Every single layer in every neural network uses matrix multiplication. This is the most important operation in all of AI.

A = torch.tensor([[1.0, 2.0, 3.0],
                   [4.0, 5.0, 6.0]])  # shape (2, 3)

B = torch.tensor([[7.0, 8.0],
                   [9.0, 10.0],
                   [11.0, 12.0]])  # shape (3, 2)

result = torch.matmul(A, B)
print(result.shape)   # torch.Size([2, 2])
# Rule: (2,3) × (3,2) → inner dims must match → result is (2,2)

Reshaping

Same numbers, different arrangement. Like rearranging 12 eggs from one row of 12 into 3 rows of 4.

x = torch.arange(12, dtype=torch.float32)
print(x.shape)           # torch.Size([12])

a = x.reshape(3, 4)     # 3 rows, 4 columns
b = x.reshape(4, -1)    # -1 means "figure it out" → (4, 3)
c = x.reshape(-1)       # flatten to 1D

# Add a dimension (needed for batch processing)
v = torch.tensor([1.0, 2.0, 3.0])   # shape (3,)
print(v.unsqueeze(0).shape)          # torch.Size([1, 3])
print(v.unsqueeze(1).shape)          # torch.Size([3, 1])

Indexing & Slicing

t = torch.tensor([[1, 2, 3],
                   [4, 5, 6],
                   [7, 8, 9]])

print(t[0, 1])     # tensor(2)       — row 0, col 1
print(t[0])        # tensor([1,2,3]) — entire first row
print(t[:, 0])     # tensor([1,4,7]) — entire first column
print(t[0:2, 1:3]) # rows 0-1, cols 1-2

CPU vs GPU — Device Management

Your computer has two processors that can do maths. The CPU is your main brain — powerful but sequential. The GPU was built for video games but turns out its parallel processing is perfect for AI.

The Standard Device Pattern

Write this at the top of every PyTorch file. It automatically uses a GPU if available, falls back to CPU if not — without changing any other code.

import torch

# Set device once — use everywhere
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using: {device}")

# Create tensor directly on device
t = torch.rand(3, 3).to(device)

# Move existing tensor to device
a = torch.tensor([1.0, 2.0])
a = a.to(device)

# Move back to CPU (required before .numpy())
a_cpu = a.cpu().numpy()

Autograd — How PyTorch Learns Automatically

This is where PyTorch becomes truly magical. The question every beginner skips: how does the model know which way to adjust?

The answer is gradients — and PyTorch computes them automatically through a system called autograd.

The Blindfolded Hiker Analogy

requires_grad, backward(), and Gradients

import torch

# Tell PyTorch to track this tensor (weights need this, not inputs)
w = torch.tensor([3.0], requires_grad=True)

# Forward pass — PyTorch secretly builds a "receipt" of every operation
y = w ** 2   # y = w²

# Backward pass — walk back through the receipt, compute gradients
y.backward()

# The gradient: dy/dw = 2w = 2×3 = 6
print(w.grad)   # tensor([6.])

The Three Sacred Lines — Always In This Order

# 1. Zero out old gradients (they accumulate if you don't!)
optimizer.zero_grad()

# 2. Compute gradients by walking backwards through the graph
loss.backward()

# 3. Update weights in the direction that reduces loss
optimizer.step()

torch.no_grad() — For Evaluation

During evaluation you don't need gradients — tracking them wastes memory and time. Disable them with:

model.eval()                          # switch to evaluation mode
with torch.no_grad():              # disable gradient tracking
    predictions = model(X_test)   # safe, efficient inference

Building Neural Networks with torch.nn

Strip away all complexity and a neural network is just: a function that takes numbers in, does a series of mathematical transformations, and produces numbers out.

The Building Blocks

nn.Module — The Base Class

Every PyTorch model inherits from nn.Module. It handles storing weights, tracking parameters, and switching between train/eval mode automatically.

import torch
import torch.nn as nn

class SimpleNetwork(nn.Module):

    def __init__(self):
        super().__init__()          # always required — activates nn.Module machinery

        self.layer1 = nn.Linear(3, 8)   # 3 inputs → 8 neurons
        self.layer2 = nn.Linear(8, 1)   # 8 → 1 output
        self.relu   = nn.ReLU()         # activation: max(0, x)

    def forward(self, x):            # defines how data flows through
        x = self.layer1(x)
        x = self.relu(x)
        x = self.layer2(x)
        return x

model = SimpleNetwork()

# Pass data through
x = torch.tensor([[1.0, 2.0, 3.0]])   # shape (1, 3) — 1 sample, 3 features
print(model(x).shape)                   # torch.Size([1, 1])

nn.Sequential — The Shortcut

model = nn.Sequential(
    nn.Linear(3, 8),
    nn.ReLU(),
    nn.Linear(8, 1)
)
# Identical result to the class above — less code for simple networks

Common Activation Functions

The Training Loop

The training loop is the heart of every PyTorch project. You will write this pattern in every single model you ever train. Memorise its structure.

Loss Functions

Optimizers

The Complete Training Loop

import torch
import torch.nn as nn
import torch.optim as optim

model     = nn.Sequential(nn.Linear(3, 8), nn.ReLU(), nn.Linear(8, 1))
loss_fn   = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

X = torch.rand(100, 3)   # 100 samples, 3 features
y = torch.rand(100, 1)   # 100 targets

for epoch in range(100):

    # ── Training phase ──────────────────────
    model.train()                  # enable training mode

    predictions = model(X)        # forward pass
    loss        = loss_fn(predictions, y)   # measure error

    optimizer.zero_grad()          # clear old gradients ← NEVER SKIP
    loss.backward()                # compute gradients
    optimizer.step()               # update weights

    # ── Evaluation phase ─────────────────────
    model.eval()
    with torch.no_grad():
        test_preds = model(X)

    if (epoch + 1) % 20 == 0:
        print(f"Epoch {epoch+1}: Loss = {loss.item():.4f}")

Dataset & DataLoader

In real AI, datasets have millions of samples — you can't load them all into memory at once. PyTorch solves this with batching: feed the model small chunks at a time.

Custom Dataset Class

Every custom Dataset must implement exactly three methods:

from torch.utils.data import Dataset, DataLoader

class MyDataset(Dataset):

    def __init__(self, X, y):
        self.X = X
        self.y = y

    def __len__(self):               # DataLoader needs total count
        return len(self.X)

    def __getitem__(self, index):   # DataLoader calls this per sample
        return self.X[index], self.y[index]

# Create dataset and dataloader
dataset    = MyDataset(X_train, y_train)
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

# Loop through batches
for X_batch, y_batch in dataloader:
    print(X_batch.shape)   # (32, features) — one batch at a time
    break

TensorDataset — The Quick Version

from torch.utils.data import TensorDataset

train_ds = TensorDataset(X_train, y_train)
test_ds  = TensorDataset(X_test,  y_test)

train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)
test_loader  = DataLoader(test_ds,  batch_size=32, shuffle=False)
# shuffle=False for test — order doesn't matter, consistency does

Saving & Loading Models

Training a model once and saving it means you never have to retrain. This is essential — deploy it, share it, resume training, or keep the best version found during training.

The State Dict Pattern

PyTorch saves the state dict — a dictionary of all weights and biases. You save numbers, not structure. Loading requires recreating the architecture first, then filling in the saved numbers.

import torch

# ── Save ──────────────────────────────────────────────────────
torch.save(model.state_dict(), "model.pth")

# ── Load ──────────────────────────────────────────────────────
loaded_model = MyModelClass()   # recreate same architecture first
loaded_model.load_state_dict(
    torch.load("model.pth", weights_only=True)
)
loaded_model.eval()             # always set eval mode after loading

Full Checkpoint — For Resuming Training

# Save checkpoint
torch.save({
    "epoch":      current_epoch,
    "model":      model.state_dict(),
    "optimizer":  optimizer.state_dict(),  # Adam has internal memory too
    "loss":       best_loss
}, "checkpoint.pth")

# Load checkpoint
ckpt = torch.load("checkpoint.pth", weights_only=False)
model.load_state_dict(ckpt["model"])
optimizer.load_state_dict(ckpt["optimizer"])
start_epoch = ckpt["epoch"]

Saving the Best Model During Training

best_loss = float("inf")   # start at infinity so any loss beats it

for epoch in range(EPOCHS):
    # ... training code ...

    if avg_loss < best_loss:
        best_loss = avg_loss
        torch.save(model.state_dict(), "best_model.pth")
        print(f"New best model saved (loss={best_loss:.4f})")

Key Functions Reference

Every important PyTorch function you need as a beginner — with purpose and usage at a glance.

Tensor Creation

Tensor Inspection & Manipulation

Autograd

Neural Network Modules

Training Utilities

Projects — Built Step by Step

Project 1: Linear Regression

The model learns to discover a hidden formula from data alone — without being told what it is.

import torch, torch.nn as nn, torch.optim as optim

torch.manual_seed(42)

# Data — model doesn't know the formula, must discover it
TRUE_WEIGHT, TRUE_BIAS = 3.0, 1.0
X     = torch.rand(100, 1)
y     = TRUE_WEIGHT * X + TRUE_BIAS + torch.randn(100, 1) * 0.1
split = 80
X_train, y_train = X[:split], y[:split]
X_test,  y_test  = X[split:], y[split:]

# Model
class LinearModel(nn.Module):
    def __init__(self): super().__init__(); self.linear = nn.Linear(1, 1)
    def forward(self, x): return self.linear(x)

model     = LinearModel()
loss_fn   = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Train
for epoch in range(200):
    model.train()
    loss = loss_fn(model(X_train), y_train)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

# Results — model discovers weight≈3.0, bias≈1.0
for name, param in model.named_parameters():
    print(f"{name}: {param.data.item():.4f}")
# linear.weight: 2.9876  — true was 3.0
# linear.bias:   1.0043  — true was 1.0

Project 2: Binary Classification

Two clusters of points. The model learns to tell them apart.

import torch, torch.nn as nn, torch.optim as optim

torch.manual_seed(42)

# Two clusters: class 0 near (1,1), class 1 near (3,3)
X = torch.cat([torch.randn(100,2)+1, torch.randn(100,2)+3])
y = torch.cat([torch.zeros(100,1), torch.ones(100,1)])
idx = torch.randperm(200)
X, y = X[idx], y[idx]
X_train, y_train = X[:160], y[:160]
X_test,  y_test  = X[160:], y[160:]

model = nn.Sequential(
    nn.Linear(2, 8), nn.ReLU(),
    nn.Linear(8, 1), nn.Sigmoid()   # outputs probability 0-1
)
loss_fn   = nn.BCELoss()             # binary cross entropy
optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(200):
    model.train()
    preds = model(X_train)
    loss  = loss_fn(preds, y_train)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

model.eval()
with torch.no_grad():
    test_preds = model(X_test)
    accuracy   = ((test_preds >= 0.5).float() == y_test).float().mean()
    print(f"Test Accuracy: {accuracy.item()*100:.1f}%")   # ~97-99%

Project 3: End-to-End with DataLoader — Student Pass/Fail

The complete, production-grade pipeline. Every best practice applied. This is the template for all real projects.

# ── Setup ─────────────────────────────────────────────────────
device = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(42)

# ── Data → Dataset → DataLoader ───────────────────────────────
train_loader = DataLoader(StudentDataset(X_train, y_train), batch_size=32, shuffle=True)
test_loader  = DataLoader(StudentDataset(X_test,  y_test),  batch_size=32, shuffle=False)

# ── Model + Loss + Optimizer ──────────────────────────────────
model     = StudentClassifier().to(device)
loss_fn   = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
best_loss = float("inf")

# ── Training Loop ─────────────────────────────────────────────
for epoch in range(30):
    model.train()
    for xb, yb in train_loader:
        pred = model(xb); loss = loss_fn(pred, yb)
        optimizer.zero_grad(); loss.backward(); optimizer.step()

    model.eval()
    with torch.no_grad():
        test_loss = sum(loss_fn(model(xb), yb) for xb,yb in test_loader) / len(test_loader)

    if test_loss < best_loss:
        best_loss = test_loss
        torch.save(model.state_dict(), "best_model.pth")

# ── Inference ─────────────────────────────────────────────────
model.load_state_dict(torch.load("best_model.pth", weights_only=True))
model.eval()
with torch.no_grad():
    prob = model(new_student_data)
    print("PASS" if prob.item() >= 0.5 else "FAIL")

Where to Go Next

Immediate Next Topics in PyTorch

• Multi-class classification — predicting from more than 2 categories with CrossEntropyLoss and Softmax
• Batch Normalisation — nn.BatchNorm1d — stabilises training of deeper networks
• Dropout — nn.Dropout(p=0.2) — prevents overfitting by randomly disabling neurons during training
• Learning rate scheduling — automatically reduce learning rate when improvement stalls

Computer Vision

• Convolutional Neural Networks (CNNs) — nn.Conv2d — the standard architecture for image tasks
• torchvision — built-in datasets (MNIST, CIFAR-10) and pretrained models
• Transfer learning — fine-tune ResNet, VGG, EfficientNet on your own data in minutes

Natural Language Processing

• Embeddings — nn.Embedding — turning words into dense numerical vectors
• Transformers — the architecture behind GPT, BERT, and all modern LLMs
• Hugging Face — fine-tune pretrained language models on custom tasks

Professional Skills

• Weights & Biases (wandb) — track experiments, visualise training curves
• Mixed precision training — torch.cuda.amp — faster training with less GPU memory
• Model deployment — TorchScript, ONNX, or FastAPI for serving predictions

What You Now Understand