PEFT & LoRA - SustainSys AI Academy

PEFT & LoRA: Parameter-Efficient Fine-Tuning for LLMs

Parameter-Efficient Fine-Tuning (PEFT) fundamentally changed LLM customization by enabling effective fine-tuning with minimal computational resources. LoRA (Low-Rank Adaptation) is the dominant technique: train only 0.01-0.1% of model parameters while maintaining or exceeding full fine-tuning quality. This enables practical LLM customization on consumer-grade hardware.

This comprehensive guide covers:

Complete mathematical foundations of low-rank decomposition and why it works theoretically
Detailed comparison of LoRA, QLoRA, Adapters, Prefix Tuning, and IA3 — when to use each
Production-ready code examples for training with PEFT library
Advanced techniques: rank selection, multi-adapter serving, adapter composition, and merging strategies
Memory analysis, GPU requirements, and hardware optimization strategies
Deployment patterns for single and multi-adapter inference systems
Real-world case studies: customer support, code generation, domain QA, personal AI
Interview preparation: 8 questions covering core concepts and practical scenarios
Comprehensive FAQ addressing 10 common questions and edge cases

Key Insight: The intrinsic dimensionality of task-specific weight updates is much lower than the full parameter space. LoRA exploits this by learning ΔW = BA where B ∈ ℝ^(m×r) and A ∈ ℝ^(r×n) with r ≪ min(m,n).

Memory Efficiency

Reduce trainable parameters from 7B to 1.3M (0.02%). Training on GPUs with 16GB vs 80GB required.

Speed

Single GPU fine-tuning in hours instead of weeks. 5-10× faster than full fine-tuning.

Modularity

Train separate task-specific adapters. Swap at inference. Serve multiple tasks from one model.

Cost

Reduce infrastructure spending by 50-100×. Democratizes LLM customization.

Why Parameter-Efficient Fine-Tuning Matters

Understanding the problem PEFT solves requires examining the full fine-tuning landscape. Large language models have grown exponentially: GPT-2 (1.5B), GPT-3 (175B), GPT-3.5 (unknown), and modern open models (LLaMA 7B-70B, Mistral, Gemma). While pre-trained models are freely available, customizing them for specific domains or tasks remained prohibitively expensive.

The Full Fine-Tuning Memory Crisis

Memory Breakdown for 7B Model Training

Model parameters (float32): 28 GB | Model parameters (float16): 14 GB | Gradients (float32): 28 GB | Optimizer momentum (float32): 28 GB | Optimizer variance (float32): 28 GB | Activation checkpoints: 3-5 GB | Total for full FT: 100+ GB. A single V100 (16GB) or even A100 (40GB) cannot handle this. You need 8× A100s, costing $12k+/month.

This barrier eliminated fine-tuning for most organizations. Startups, academic labs, and individual researchers had two choices:

Use pre-trained models without customization (suboptimal quality)
Build from scratch (years of development, massive datasets required)

LoRA solved this constraint. Microsoft researchers (Hu et al. 2021) hypothesized that fine-tuning weight updates are intrinsically low-rank. They empirically verified this across multiple models and datasets, showing that rank-8 LoRA adapters achieved comparable performance to full fine-tuning on NLU tasks, NLG tasks, and instruction-following.

Quantified Impact

Dimension	Full FT	LoRA	Improvement
GPU Memory	80-100 GB	14-32 GB	60-85% reduction
Trainable Params	7,000M (100%)	1.3M (0.02%)	5400× reduction
Training Time (1 epoch)	8-12 hours	1-2 hours	5-10× speedup
Adapter Size	14,000 MB	50-100 MB	140-280× smaller
Hardware Cost	8× A100 (~$12k/mo)	1× A100 (~$1.5k/mo)	8× cost reduction
Turnaround Time	1-2 weeks	1-2 days	7-10× faster
Model Quality	100%	99%	Negligible loss

The LoRA Breakthrough: On GLUE benchmark, a rank-8 LoRA adapter on RoBERTa achieved 99.1% of full fine-tuning quality while using 0.02% of trainable parameters. This result was surprising and opened a new research direction.

Downstream Impact on AI Development

Democratization

Thousands of practitioners now fine-tune models who couldn't afford $10k+ infrastructure before.

Acceleration

PEFT research exploded post-2021. New methods (adapters, IA3, prefix tuning) emerged because fine-tuning was now viable.

Market Competition

Open LLM ecosystem thrives. Without PEFT, only mega-corporations could customize models. PEFT enabled open models to compete.

Production ML

Companies deploy task-specific adapters instead of model zoos. Smaller serving infrastructure, versioning, and model management.

PEFT Fundamentals: The Complete Landscape

PEFT is an umbrella term for techniques that keep most of a pre-trained model frozen while training a small adapter layer. This simple principle has profound implications for efficiency and quality.

PEFT Mathematical Principle

y = f_base(x) + f_adapter(x)

Where:
f_base(x) is the frozen pre-trained model
f_adapter(x) is a small learnable component

The adapter is typically 0.01-2% of the base model size.
During training: ∂L/∂base = 0 (frozen)
During training: ∂L/∂adapter ≠ 0 (optimized)

Why This Works

Feature Reuse: Pre-training already learned excellent general representations. Adaptation just needs to steer these representations toward the specific task.
Intrinsic Dimensionality: Task adaptation lies in a low-dimensional subspace of the parameter space. You don't need billions of degrees of freedom — hundreds of thousands suffice.
Catastrophic Forgetting Mitigation: By freezing the base model, PEFT preserves knowledge from pre-training while adapting to new domains. Full fine-tuning risks forgetting general knowledge.
Computational Efficiency: Smaller adapters reduce memory (both weights and gradients/optimizer states), enable batching on consumer hardware, and reduce training time.

PEFT Method Comparison Matrix

Method	Mechanism	Params %	Pros	Cons
LoRA	Low-rank weight decomposition	0.01-0.1%	Excellent quality, simple, widely supported	Assumes low-rank, cannot change architecture
QLoRA	LoRA + 4-bit base quantization	0.01-0.1%	Extreme memory savings, same quality	Slower training (quantization overhead)
Adapters	Bottleneck layers in transformer blocks	0.5-2%	Composable, task routing possible	Slower inference, larger than LoRA
Prefix Tuning	Learnable soft prompt tokens	0.1-1%	Interpretable, few-shot friendly	Reduces sequence length, lower quality
IA3	Element-wise activation scaling	0.001%	Ultra-efficient, minimal computation	Lower final quality, less research support
BitFit	Only train bias vectors	0.0009%	Simplest, barely any parameters	Quality trails other methods

Feature Visualization Insight

Research using neural network visualization shows that LoRA adapters learn task-specific rotations and scalings of pre-trained feature representations. The base model's learned features (attention to language structure, semantic relationships) are preserved. The adapter amplifies task-relevant features and suppresses task-irrelevant ones.

LoRA: Low-Rank Adaptation — Deep Dive

LoRA (Hu et al., 2021) is the most successful PEFT method. The core idea is elegant: instead of fine-tuning weight matrix W, decompose the update as a product of two small matrices.

LoRA Mathematical Definition

Standard forward pass: y = Wx

With LoRA adaptation: y = Wx + \Delta Wx = Wx + BAx

Where:
W ∈ ℝ^(m×n) is the pre-trained weight matrix (frozen)
B ∈ ℝ^(m×r) is the down-projection matrix (trainable)
A ∈ ℝ^(r×n) is the up-projection matrix (trainable)
r ≪ min(m, n) is the rank (typically 4-64)

Parameter reduction:
Full: m·n parameters
LoRA: r·(m + n) parameters

For a 1000×1000 weight matrix with r=8:
Full: 1,000,000 parameters
LoRA: 8×2000 = 16,000 parameters
Reduction: 62.5× fewer parameters

Initialization Strategy

Python — Correct LoRA Initialization
import torch.nn as nn
import torch

# CORRECT initialization (as in Hu et al. 2021)
lora_a = nn.Linear(in_features, rank, bias=False)
lora_b = nn.Linear(rank, out_features, bias=False)

# Initialize A with Gaussian noise (important for gradient flow!)
nn.init.normal_(lora_a.weight, mean=0, std=1/rank)

# Initialize B to zero (so ΔW = BA starts at zero, no initial disruption)
nn.init.zeros_(lora_b.weight)

# INCORRECT (common beginner mistake)
# Both randomly initialized → large initial update, training instability
# nn.init.normal_(lora_b.weight)  # DON'T DO THIS!

Why Zero Initialization for B?: At the start of training, LoRA should contribute zero to the forward pass (ΔW = 0). This ensures the model doesn't suddenly change behavior. Gradients through B start from random, and A is initialized to provide gradient signal. This is critical for stable training.

Scaling Factor: α/r

LoRA includes a scaling factor α/r applied to the update. This is crucial for hyperparameter transfer.

LoRA Scaling Analysis

Update magnitude: y = Wx + (α/r) × BAx

If α = r (typical choice):
Scaling factor = r/r = 1 (constant!)

Without scaling (α = 1):
For r=4: scaling = 1/4 = 0.25
For r=8: scaling = 1/8 = 0.125
For r=16: scaling = 1/16 = 0.0625
← Different learning rates for different ranks!

With α = r:
For any r, scaling ≈ 1
← Same learning dynamics regardless of rank

Consequence: You can change rank without retuning hyperparameters!

Where to Apply LoRA?

Not all weight matrices benefit equally from LoRA. Research shows:

Layer Type	Impact	Typical Strategy	Why
Query, Key, Value	Very High	Always apply LoRA	These projections determine what information each token attends to. Task-specific, high flexibility.
Output Projection	High	Almost always apply	Combines attention heads. Task-specific routing of information.
FFN Up (expansion)	Moderate-High	Apply in ~60% of cases	Maps to intermediate dimension. Some task-specificity.
FFN Down (projection)	Moderate	Apply in ~40% of cases	Projects back to model dimension. Less essential than up.
Token Embeddings	Very Low	Skip unless vocab changes	Pre-trained embeddings are general-purpose. Rarely need task-specific adjustment.
Layer Norms	Negligible	Always frozen	Normalization is task-agnostic. Training breaks stability.
Biases	Low	Usually frozen	Biases are low-capacity. Include only with large datasets.

Empirical Finding: Focus on Attention

In practice, most high-quality results come from applying LoRA only to attention projections (Q, K, V, O). FFN layers help but with diminishing returns. Embeddings rarely need LoRA.

LoRA Extensions and Variants

LoRA+: Uses different learning rates for B and A. Empirically, A benefits from higher learning rate than B. This provides modest quality improvements.

DoRA (Decomposed LoRA): Separates weight matrices into magnitude and direction components. Fine-tunes magnitude separately from direction via LoRA. Better quality on some benchmarks but more complex.

Sparse LoRA: Applies structured sparsity to B and A matrices. Many values are exactly zero, further reducing computation and memory. Trade-off: slightly lower quality.

Mixture of LoRAs (MoLoRA): Multiple LoRA adapters with attention-based gating. Enables soft composition of multiple adapters. Useful for multi-task learning within a single forward pass.

QLoRA: Quantized LoRA — Extreme Efficiency

QLoRA (Dettmers et al., 2023) combines LoRA with aggressive 4-bit base model quantization. This enables fine-tuning of 70B-parameter models on 40GB GPUs — previously impossible.

The Quantization Strategy

4-bit Quantization Process

Original weights W ∈ ℝ^(m×n) (float32 or float16)

1. Compute statistics: min_val, max_val, range
2. Map to integers: int_val = round((W - min_val) / (range / 15)) ∈ [0, 15]
3. Store 4 bits per value (0.5 bytes vs 2-4 bytes in float32)
4. Dequantize during forward: W_dequant = int_val × (range/15) + min_val

Memory savings: 8-16× reduction for base model weights

NF4 (Normal Float 4) variant:
- Quantization levels are information-theoretically optimal for Gaussian dist
- Uses 16 specific floating-point values (not uniform integer mapping)
- Preserves more information for neural network weights

QLoRA Training Dynamics

Unlike simply quantizing and fine-tuning, QLoRA carefully manages precision:

Base Model: Quantized to 4-bit NF4 for storage and forward pass
Adapter (LoRA): Kept in full precision (float32) throughout training
Forward Pass: Base weights dequantized to float32, used to compute hidden states
Adapter Pass: LoRA adapter computes in full precision
Backward Pass: Gradients flow only through adapter, not through quantized base (avoids quantization-induced gradient issues)

Python — QLoRA Implementation with bitsandbytes

from transformers import BitsAndBytesConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
import torch

# Step 1: Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,      # Double quantization (quantize scales too)
    bnb_4bit_quant_type="nf4",            # NF4 (normal float 4) quantization
    bnb_4bit_compute_dtype=torch.bfloat16 # Compute in bfloat16
)

# Step 2: Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto"
)

# Step 3: Apply LoRA on quantized base
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# Prepare for training
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)

print(model)
# Now memory usage is ~3.5-4 GB even for 7B models!

QLoRA Quality-Efficiency Tradeoff

Metric	Standard LoRA	QLoRA
Base Model	float16 (14GB for 7B)	4-bit NF4 (3.5GB for 7B)
Training Speed	1.0× baseline	0.7-0.8× (quantization/dequantization overhead)
Training Memory	~14-16 GB	~3.5-4 GB
Inference Speed	1.0× (if merged)	0.95-1.0× (minimal dequantization cost per token)
Final Quality	99-100%	97-99% (small but measurable degradation)
Recommended Use	16GB+ VRAM	8-16GB VRAM (memory-constrained)

The QLoRA Breakthrough: A single 48GB GPU could fine-tune 65B LLaMA models. This was revolutionary — suddenly, affordable hardware (~$8-10k) could handle models that previously required enterprise infrastructure ($100k+). This enabled a new wave of open-source fine-tuning research.

Gradient Precision in QLoRA

A subtle but important detail: even though base weights are 4-bit, gradients are computed in full precision. Here's why:

Python — QLoRA Precision Management
# Base model weights: 4-bit quantized
W_quantized = 4_bit_tensor  # 0.5 bytes per element

# During forward pass, dequantize to float32
W_float32 = dequantize(W_quantized)  # Temporary, only during forward

# Compute activations
activations = W_float32 @ input

# Gradients through LoRA adapter (full precision)
grad_lora_down = activations.T @ grad_output
grad_lora_up = grad_output @ hidden_states.T

# Note: No gradients for W_float32 (base model frozen)
# Quantization error does NOT accumulate through gradients!

# This is key: quantization noise is isolated in forward pass
# It does not degrade gradient signal for the adapter

Other Parameter-Efficient Methods

Adapters (Bottleneck Adapters)

Adapter Architecture

For each transformer layer:

Adapter(x) = W_up(σ(W_down(x + residual)))

Dimensions:
Input x: d_model (e.g., 4096)
W_down: d_model → r (e.g., 4096 → 256, ratio 1/16)
σ: Activation (ReLU or GELU)
W_up: r → d_model (e.g., 256 → 4096)

Per-adapter params:
W_down: 4096 × 256 = 1,048,576
W_up: 256 × 4096 = 1,048,576
Total: ~2M per adapter

For 32-layer model: 64M parameters total (0.9% for 7B model)

Key Difference from LoRA: Adapters are inserted as separate layers, not fused into weight matrices. This enables easy removal, stacking, and composition.

Advantages:

Task Composition: Stack multiple adapters sequentially
Adapter Merging: Combine multiple task adapters into one
Routing: Dynamically select which adapter based on input
Layer-specific tuning: Different reduction ratios for different layers

Disadvantages:

Slower inference (each adapter adds computation)
Larger than LoRA (0.5-2% vs 0.01-0.1%)
Less widely supported in production libraries
Requires architectural integration during model loading

Prefix Tuning

Prefix Tuning Mechanism

Prepend learnable prefix tokens to every layer's KV cache:

For each transformer layer:
prefix_k = P_k ∈ ℝ^(prefix_len × d_k) // learnable
prefix_v = P_v ∈ ℝ^(prefix_len × d_v) // learnable

actual_k = [prefix_k; cached_k] // concatenate
actual_v = [prefix_v; cached_v]

attention = softmax(Q · actual_k^T) · actual_v

Total parameters:
num_layers × prefix_len × (d_k + d_v)
For 32 layers, prefix_len=100, d_k=d_v=64:
32 × 100 × 128 = 409,600 params (~0.006% for 7B model)

Interpretation: The prefix acts like a soft, continuous prompt that influences the model's behavior throughout the network. Unlike hard prompts (discrete tokens), prefix vectors are learned end-to-end.

Strengths:

Interpretable: Prefix vectors capture task intent
Very small parameter count (0.001-0.1%)
Effective for few-shot and prompt-based tasks
Can mix multiple prefixes (ensemble)

Weaknesses:

Consumes input sequence length (reduces tokens for actual content)
Performance often trails LoRA and Adapters
Learning can be unstable (manifold of equivalent prefixes)
Limited to prompt-based adaptation (doesn't work for ranking/classification well)

IA3: Information-preserving and parameter-efficient adaptation

IA3 Mechanism (Element-wise Scaling)

For each activation in FFN and attention:

For attention: v_out = Attention(Q, K, V) ⊙ s_attn
For FFN up: hidden = W_up(x) ⊙ s_ffn_up
For FFN down: out = W_down(hidden) ⊙ s_ffn_down

Where ⊙ is element-wise multiplication, and s_* are learnable scalars.

Total parameters:
~ 2 × d_model + 2 × d_ff per layer
For 32 layers: 32 × (2×4096 + 2×11008) = ~961,536 params
That is 0.01% for 7B model!

IA3 is Extremely Efficient: Only element-wise scalar multiplication. Minimal computational overhead. Surprisingly effective — maintains 92-95% of full FT quality with 0.01% parameters.

Mathematical Foundation & Theory

Singular Value Decomposition (SVD) Theory

Complete SVD Theorem

Any matrix W ∈ ℝ^(m×n) can be decomposed as:

W = U Σ V^T

Where:
U ∈ ℝ^(m×m): orthonormal matrix (left singular vectors)
Σ ∈ ℝ^(m×n): diagonal matrix with singular values σ₁ ≥ σ₂ ≥ ... ≥ σₙ ≥ 0
V ∈ ℝ^(n×n): orthonormal matrix (right singular vectors)

The Frobenius norm of a rank-r approximation error:
||W - W_r||_F = √(Σᵢ₌ᵣ₊₁ σᵢ²)

Eckart-Young Theorem: The truncated SVD using the top-r singular values
minimizes ||W - W_r||_F among all rank-r matrices.

Why Weight Updates Are Low-Rank

During pre-training, large models learn to map diverse inputs to rich representations. These representations capture linguistic structure, semantic meaning, world knowledge, and reasoning patterns. Fine-tuning does not need to relearn all of this.

Empirical Evidence (Hu et al. 2021): When computing the SVD of ΔW = W_fine_tuned - W_pretrained across various models and tasks, the singular values decay rapidly:

Python — SVD Analysis of Weight Updates
import numpy as np
from scipy.linalg import svd

# Example: Hypothetical weight update ΔW
delta_W = np.random.randn(4096, 4096) * 0.001

# Compute SVD
U, singular_values, Vt = svd(delta_W, full_matrices=False)

# Analyze decay
cumsum = np.cumsum(singular_values ** 2)
cumsum = cumsum / cumsum[-1]  # Normalize

# How many singular values needed for 90% variance?
rank_90 = np.argmax(cumsum >= 0.9) + 1
rank_95 = np.argmax(cumsum >= 0.95) + 1
rank_99 = np.argmax(cumsum >= 0.99) + 1

print(f"Rank for 90% variance: {rank_90}")  # Often ~4-8
print(f"Rank for 95% variance: {rank_95}")  # Often ~8-16
print(f"Rank for 99% variance: {rank_99}")  # Often ~16-32

# Visualization
import matplotlib.pyplot as plt
plt.plot(cumsum)
plt.axhline(y=0.95, color='r', linestyle='--', label='95% threshold')
plt.xlabel('Rank')
plt.ylabel('Cumulative Variance Explained')
plt.title('Singular Value Spectrum of Weight Updates')
plt.legend()
plt.show()

Gradient Flow Analysis

Gradient Computation Through LoRA

Forward pass: y = Wx + BAx

Loss: L = loss_fn(y, target)

Backward pass:
∂L/∂(BAx) = ∂L/∂y (chain rule)
∂L/∂(BAx) = ∂L/∂y · ∂y/∂(BAx)

∂L/∂A: (∂L/∂(BAx)) · ∂(BAx)/∂A = (∂L/∂(BAx)) · B^T x^T
∂L/∂B: (∂L/∂(BAx)) · ∂(BAx)/∂B = (∂L/∂(BAx)) · x A

∂L/∂W = ∂L/∂y (but W is frozen, so no update)

Key insight: Gradient ∂L/∂(BAx) is shared between B and A.
This creates a bottleneck: B and A must cooperatively represent the gradient.

Benefit: The low-rank bottleneck can improve gradient conditioning. Instead of optimizing in a billion-dimensional space, you optimize in a rank-limited space. This often leads to better generalization and faster convergence.

Architecture & Design Considerations

Rank Selection Strategy

Rank Selection Heuristics

Empirical Guidelines (from literature):

1. Task-agnostic default: r=8
Works well for most use cases (90%+)

2. Model size scaling:
1-7B models: r=4-8
7-30B models: r=8-16
30B+ models: r=16-32

3. Dataset size scaling:
Small (≤1k): r=4
Medium (1k-10k): r=8
Large (10k-100k): r=16
Extra-large (100k+): r=32-64

4. Complexity of task:
Simple classification: r=4
General instruction-tuning: r=8
Complex reasoning: r=16-32

5. Quality requirements:
High-quality (95%+ of FT): r=16-32
Good quality (90%+ of FT): r=8
Fast prototyping: r=4

The 80/20 Rule: 80% of adaptation capacity comes from the top 4-8 singular directions. Diminishing returns for r > 16 in most tasks.

Learning Rate Dynamics

Aspect	Full FT	LoRA	QLoRA
Learning Rate	1e-5 to 5e-5	1e-4 to 5e-4	1e-4 to 5e-4
Warmup Steps	500-1000 (helpful)	0-100 (optional)	0-100 (optional)
Learning Rate Schedule	Cosine decay (common)	Cosine or constant	Constant or cosine
Gradient Clipping	1.0 (often helpful)	Usually not needed	Usually not needed
Weight Decay	0.01	0.01	0.01

Why Higher LR for LoRA?

LoRA operates in a smaller, lower-dimensional parameter space. The Hessian (curvature) is typically better-conditioned. Higher learning rates are stable without divergence. This enables faster convergence and shorter training.

Target Module Configuration

Python — PEFT Target Module Selection Patterns
from peft import LoraConfig

# Pattern 1: Minimal (Q and V only)
config_minimal = LoraConfig(
    target_modules=["q_proj", "v_proj"],
    r=8, lora_alpha=16
)
# Useful for: Quick experiments, memory-constrained, CPU inference
# Quality: 95-98% of full FT

# Pattern 2: Standard (Q, K, V, O in attention)
config_standard = LoraConfig(
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    r=8, lora_alpha=16
)
# Useful for: Most use cases, balanced quality/efficiency
# Quality: 98-99% of full FT

# Pattern 3: Comprehensive (attention + FFN)
config_comprehensive = LoraConfig(
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",      # Attention
        "gate_proj", "up_proj", "down_proj"          # FFN (LLaMA style)
    ],
    r=8, lora_alpha=16
)
# Useful for: High-quality needed, memory available, time not critical
# Quality: 99-100% of full FT

# For different models, module names vary:
# LLaMA/Mistral: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
# GPT-2/3: c_attn (all QKV), c_proj, c_fc, c_proj
# BERT: query, key, value, dense
# Use model.named_modules() to inspect actual layer names!

Implementation Guide: Step-by-Step

Python — Complete LoRA Fine-Tuning Pipeline
#!/usr/bin/env python3
from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from datasets import load_dataset
import torch
import os

# ===== CONFIGURATION =====
MODEL_NAME = "meta-llama/Llama-2-7b-hf"
OUTPUT_DIR = "./llama-7b-custom-lora"
DATASET_NAME = "tatsu-lab/alpaca"  # Or your custom dataset

# ===== STEP 1: LOAD BASE MODEL =====
print("Loading base model...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

# Ensure padding token is set
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# ===== STEP 2: APPLY LORA =====
print("Applying LoRA...")
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# ===== STEP 3: LOAD AND PREPARE DATA =====
print("Loading dataset...")
dataset = load_dataset(DATASET_NAME)

def preprocess_function(examples):
    # Format: Instruction - Input - Output (alpaca format)
    texts = []
    for instruction, inp, output in zip(
        examples.get("instruction", []),
        examples.get("input", []),
        examples.get("output", [])
    ):
        text = f"Instruction: {instruction}\nInput: {inp}\nOutput: {output}"
        texts.append(text)
    
    tokenized = tokenizer(
        texts,
        truncation=True,
        max_length=512,
        padding="max_length",
        return_tensors=None
    )
    
    # For causal LM, labels = input_ids
    tokenized["labels"] = tokenized["input_ids"].copy()
    return tokenized

# Process dataset
train_dataset = dataset["train"].map(
    preprocess_function,
    batched=True,
    remove_columns=["instruction", "input", "output", "text"]
)

# ===== STEP 4: TRAINING ARGUMENTS =====
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    warmup_steps=100,
    weight_decay=0.01,
    logging_steps=10,
    learning_rate=5e-4,
    lr_scheduler_type="cosine",
    save_strategy="epoch",
    save_total_limit=2,
    gradient_accumulation_steps=1,
    gradient_checkpointing=False,
    fp16=True,
    seed=42,
)

# ===== STEP 5: CREATE TRAINER =====
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

# ===== STEP 6: TRAIN =====
print("Starting training...")
trainer.train()

# ===== STEP 7: SAVE ADAPTER =====
print(f"Saving adapter to {OUTPUT_DIR}...")
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

print("✓ Training complete!")

Memory-Optimized Training for Constrained GPUs

Python — Memory Optimization Techniques
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import BitsAndBytesConfig
import torch

# For 8GB GPU (aggressive optimization)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16
    ),
    device_map="auto"
)

# Apply LoRA
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
model = get_peft_model(model, lora_config)
model = prepare_model_for_kbit_training(model)

# Training args for memory efficiency
training_args = TrainingArguments(
    per_device_train_batch_size=1,      # Minimal batch size
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=4,      # Simulate batch_size=4
    gradient_checkpointing=True,        # Trade compute for memory
    max_grad_norm=0.3,                  # Gradient clipping
    warmup_ratio=0.03,
    lr_scheduler_type="linear",
    optim="paged_adamw_8bit",           # Memory-efficient optimizer
    num_train_epochs=1,
)

# Result: 4GB training possible for 7B model!

Advanced Code Examples

QLoRA with Gradient Checkpointing

Python — QLoRA with Extreme Optimization
from transformers import BitsAndBytesConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# Ultra-efficient configuration for 70B model on 40GB A100
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    quantization_config=bnb_config,
    device_map="auto",
    load_in_8bit_fp32_cpu_offload=True  # Offload some ops to CPU
)

lora_config = LoraConfig(r=8, lora_alpha=16, target_modules=[...])
model = get_peft_model(model, lora_config)
model = prepare_model_for_kbit_training(model)

# Set up training with aggressive memory saving
model.config.use_cache = False  # Disable KV cache during training
model.gradient_checkpointing_enable()  # Store only some activations

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        gradient_checkpointing=True,
        max_steps=1000,
        learning_rate=1e-4,
        optim="paged_adamw_8bit"
    ),
    train_dataset=train_dataset
)

trainer.train()

Merging and Unloading Adapters

Python — Adapter Merging for Production

from peft import PeftModel, AutoPeftModelForCausalLM from transformers import AutoTokenizer # Load base model base_model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", torch_dtype=torch.float16, device_map="auto" ) # Load adapter peft_model = PeftModel.from_pretrained( base_model, "llama-7b-custom-lora" ) # Merge LoRA weights into base model # This bakes the low-rank update into the weight matrices merged_model = peft_model.merge_and_unload() # Save merged model (now a standard HuggingFace model) merged_model.save_pretrained("llama-7b-merged") tokenizer.save_pretrained("llama-7b-merged") # Use merged model anywhere without PEFT library! from transformers import pipeline pipe = pipeline("text-generation", model="llama-7b-merged", tokenizer="llama-7b-merged") output = pipe("Explain transformers:", max_length=100)

Multi-Adapter Serving at Scale

Python — Serving Multiple Adapters from One Model
from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load base model once
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load multiple task-specific adapters
adapters = {
    "customer_support": "adapter-support-lora",
    "technical_qa": "adapter-technical-lora",
    "code_generation": "adapter-code-lora",
    "creative_writing": "adapter-creative-lora"
}

# Load all adapters into memory
model = base_model
for task_name, adapter_path in adapters.items():
    model = PeftModel.from_pretrained(
        model,
        adapter_path,
        adapter_name=task_name  # Name for this adapter
    )

# Inference function with task routing
def generate_with_task(prompt, task_name, max_tokens=100):
    """Generate with task-specific adapter"""
    
    # Switch to appropriate adapter
    model.set_adapter(task_name)
    
    # Tokenize input
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    
    # Generate
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_length=max_tokens,
            temperature=0.7,
            top_p=0.9,
            do_sample=True
        )
    
    # Decode output
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

# Example usage
support_response = generate_with_task(
    "How do I reset my password?",
    "customer_support"
)
code_response = generate_with_task(
    "Write Python to sort a list",
    "code_generation"
)
story = generate_with_task(
    "Once upon a time...",
    "creative_writing"
)

print("Support:", support_response)
print("Code:", code_response)
print("Story:", story)

Custom LoRA Training from Scratch

Python — Implementing LoRA without PEFT (Educational)
import torch
import torch.nn as nn
import torch.nn.functional as F

class LoRALayer(nn.Module):
    """Simple LoRA adapter for a linear layer"""
    
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.rank = rank
        self.alpha = alpha
        self.scaling = alpha / rank
        
        # Trainable LoRA matrices
        self.lora_down = nn.Linear(in_features, rank, bias=False)
        self.lora_up = nn.Linear(rank, out_features, bias=False)
        
        # Initialization (critical!)
        nn.init.kaiming_uniform_(self.lora_down.weight, a=5**0.5)
        nn.init.zeros_(self.lora_up.weight)
    
    def forward(self, x):
        # LoRA computation: (α/r) × (W_up × W_down × x)
        lora_out = self.lora_down(x)
        lora_out = self.lora_up(lora_out)
        return lora_out * self.scaling

class LinearWithLoRA(nn.Module):
    """Linear layer with LoRA adapter"""
    
    def __init__(self, base_linear, rank=8):
        super().__init__()
        self.base_linear = base_linear
        self.lora = LoRALayer(
            base_linear.in_features,
            base_linear.out_features,
            rank=rank
        )
    
    def forward(self, x):
        # y = W·x + LoRA(x)
        base_out = self.base_linear(x)
        lora_out = self.lora(x)
        return base_out + lora_out

# Integration example
class TransformerBlockWithLoRA(nn.Module):
    def __init__(self, base_block, apply_lora_to_qkv=True, apply_lora_to_ffn=False):
        super().__init__()
        self.base_block = base_block
        
        if apply_lora_to_qkv:
            # Wrap attention projections
            self.q_proj_lora = LinearWithLoRA(base_block.self_attn.q_proj, rank=8)
            self.v_proj_lora = LinearWithLoRA(base_block.self_attn.v_proj, rank=8)
            self.k_proj_lora = LinearWithLoRA(base_block.self_attn.k_proj, rank=8)
            self.o_proj_lora = LinearWithLoRA(base_block.self_attn.o_proj, rank=8)
            
            # Freeze base
            for param in base_block.self_attn.parameters():
                param.requires_grad = False
            # Only LoRA params are trainable
    
    def forward(self, hidden_states, attention_mask=None):
        # Forward through base block, but use LoRA-enhanced projections
        # (simplified for illustration)
        return self.base_block(hidden_states, attention_mask)

Rank Analysis and Selection

Python — Finding Optimal Rank for Your Task
import numpy as np
import matplotlib.pyplot as plt
from peft import LoraConfig, get_peft_model
from transformers import Trainer, TrainingArguments

# Grid search over ranks
ranks_to_test = [2, 4, 8, 16, 32, 64]
results = {rank: {"train_loss": [], "eval_loss": None, "params": 0} for rank in ranks_to_test}

for rank in ranks_to_test:
    print(f"\nTraining with rank={rank}...")
    
    # Load fresh model for each rank
    model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", ...)
    
    # Configure LoRA with this rank
    config = LoraConfig(
        r=rank,
        lora_alpha=16,
        target_modules=["q_proj", "v_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM"
    )
    
    model = get_peft_model(model, config)
    
    # Count trainable params
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    results[rank]["params"] = trainable
    
    # Train (abbreviated)
    trainer = Trainer(...)
    trainer.train()
    
    # Evaluate
    eval_results = trainer.evaluate()
    results[rank]["eval_loss"] = eval_results["eval_loss"]
    results[rank]["train_loss"] = eval_results.get("train_loss", 0)

# Analysis
print("\n" + "="*60)
print("RANK ANALYSIS RESULTS")
print("="*60)
for rank in ranks_to_test:
    loss = results[rank]["eval_loss"]
    params_m = results[rank]["params"] / 1e6
    print(f"Rank {rank:2d}: Loss={loss:.4f}  Params={params_m:6.1f}M")

# Plot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

losses = [results[r]["eval_loss"] for r in ranks_to_test]
params = [results[r]["params"]/1e6 for r in ranks_to_test]

# Plot 1: Loss vs Rank
ax1.plot(ranks_to_test, losses, 'o-', linewidth=2, markersize=8)
ax1.set_xlabel("LoRA Rank", fontsize=12)
ax1.set_ylabel("Validation Loss", fontsize=12)
ax1.set_title("Loss vs Rank", fontsize=14)
ax1.grid(True, alpha=0.3)

# Plot 2: Loss vs Parameters (Pareto frontier)
colors = ['green' if l < min(losses)*1.01 else 'blue' for l in losses]
ax2.scatter(params, losses, s=200, c=colors, alpha=0.6)
for i, rank in enumerate(ranks_to_test):
    ax2.annotate(f"r={rank}", (params[i], losses[i]), fontsize=10, ha="center")
ax2.set_xlabel("Trainable Parameters (Millions)", fontsize=12)
ax2.set_ylabel("Validation Loss", fontsize=12)
ax2.set_title("Quality vs Efficiency Tradeoff", fontsize=14)
ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig("rank_analysis.png", dpi=150)
print("\n✓ Saved rank analysis plot to rank_analysis.png")

Comprehensive Method Comparison

Metric	Full FT	LoRA	QLoRA	Adapters	Prefix	IA3
Trainable %	100%	0.1%	0.1%	1%	0.5%	0.01%
Memory (7B)	~80GB	~14GB	~4GB	~16GB	~15GB	~14GB
Train Speed	1.0×	0.95×	0.7×	0.85×	0.9×	0.98×
Quality (%)	100%	99%	97%	98%	95%	92%
Inference	1.0×	1.0×† (merged)	1.0×	0.95×	0.98×	1.0×
Min GPU	80GB	24GB	8GB	32GB	24GB	24GB
Adapter Size	N/A	50-100MB	50-100MB	500MB-2GB	100-500MB	100-500MB
Composability	N/A	Fair	Fair	Excellent	Good	Limited
Best For	Unlimited compute	Default choice	Tight memory	Multi-task	Few-shot	Extreme efficiency

† Inference speed identical to base if adapter is merged into weights. Slight overhead (1-2%) if loaded separately.

Visual Comparison: Memory and Parameters

Memory Usage Comparison (7B Model Training)

Full FT

104 GB

LoRA

14 GB

QLoRA

4 GB

Trainable Parameters Comparison

Full FT (7B)

7000M

LoRA (r=8)

10M

Adapters

70M

Memory & Parameter Detailed Analysis

Complete Memory Accounting

Total Memory Formula

M_total = M_model + M_gradients + M_optimizer + M_activations

For 7B model (float16) with Adam optimizer:

M_model = 7B × 2 bytes = 14 GB
M_gradients = 7B × 2 bytes = 14 GB (only trainable params)
M_optimizer = 7B × (4 + 4) bytes = 56 GB (momentum + variance)
M_activations = batch×seq×hidden×layers × 2 ≈ 3-5 GB

Full FT Total ≈ 14 + 14 + 56 + 5 = 89 GB

With LoRA (1.3M trainable):
M_gradients = 1.3M × 2 bytes = 2.6 MB
M_optimizer = 1.3M × 8 bytes = 10.4 MB

LoRA Total ≈ 14 + 0.01 + 0.01 + 5 = 19 GB (or 31 GB with safety margin)

Memory Reduction Strategies

Strategy	Memory Reduction	Speed Impact	Quality Impact	Difficulty
Mixed Precision (fp16)	50%	Faster!	None	Trivial
Gradient Checkpointing	30-50%	-20% speed	None	Easy
LoRA	85%	-5% speed	-1% quality	Easy
4-bit Quantization	75% (model)	-30% speed	-3% quality	Moderate
Batch Size ≤ 2	Variable	Slower convergence	+2-5% loss	Trivial

Recommended Stack: Always do mixed precision (free). Then LoRA (massive savings). If still tight, add gradient checkpointing. Only then consider quantization.

Python — Memory Estimation Tool
def estimate_training_memory(
    model_params_b: float,
    sequence_length: int = 512,
    batch_size: int = 8,
    use_lora: bool = False,
    use_quantization: bool = False,
    gradient_checkpointing: bool = False,
    precision: str = "fp16"  # fp16, fp32, or 4bit
):
    """Estimate GPU memory needed for training"""
    
    # Model weights
    if use_quantization:
        model_mem = (model_params_b * 1e9 * 0.5) / 1e9  # 4-bit: 0.5 bytes/param
    elif precision == "fp16":
        model_mem = (model_params_b * 1e9 * 2) / 1e9
    else:  # fp32
        model_mem = (model_params_b * 1e9 * 4) / 1e9
    
    # Trainable parameters
    if use_lora:
        # Estimate: ~0.3% of parameters for LoRA
        trainable_params = model_params_b * 1e9 * 0.003
    else:
        trainable_params = model_params_b * 1e9
    
    # Gradient memory
    grad_mem = (trainable_params * 4) / 1e9  # Gradients usually float32
    
    # Optimizer states (Adam: momentum + variance)
    optimizer_mem = (trainable_params * 8) / 1e9
    
    # Activation memory
    if gradient_checkpointing:
        # Only store O(sqrt(layers)) activations
        activation_mem = 1.0  # Rough estimate
    else:
        # Store all activations
        hidden_size = int((model_params_b * 1e9 / 12 / sequence_length) ** 0.5)  # Rough
        activation_mem = (batch_size * sequence_length * hidden_size * 4) / 1e9  # Bytes
    
    total = model_mem + grad_mem + optimizer_mem + activation_mem
    
    return {
        "model": model_mem,
        "gradients": grad_mem,
        "optimizer": optimizer_mem,
        "activations": activation_mem,
        "total": total
    }

# Examples
print("Memory Requirements for Training:")
print("="*70)

configs = [
    ("Full FT 7B (fp16)", dict(model_params_b=7, use_lora=False, precision="fp16")),
    ("Full FT 7B (fp32)", dict(model_params_b=7, use_lora=False, precision="fp32")),
    ("LoRA 7B", dict(model_params_b=7, use_lora=True, precision="fp16")),
    ("LoRA 7B + GradCP", dict(model_params_b=7, use_lora=True, gradient_checkpointing=True)),
    ("QLoRA 7B", dict(model_params_b=7, use_lora=True, use_quantization=True)),
]

for name, cfg in configs:
    est = estimate_training_memory(**cfg)
    print(f"{name:25s}: {est['total']:6.1f} GB")

Practical Deployment & Serving

Merging vs Runtime Loading

Strategy 1: Merge and Deploy

Pros: No framework dependencies, standard HF model, no latency overhead
Cons: Lose adapter modularity, larger model size, single task
Best for: Production deployment, speed critical, single well-defined task

Strategy 2: Runtime Loading

Pros: Multiple adapters from one base, easy A/B testing, modular
Cons: Framework dependency (PEFT), slight latency (adapter loading), complex serving
Best for: Multi-task systems, frequent adapter updates, research/experimentation

Production Deployment Checklist

Before Deploying Adapters

Test adapter on held-out test set (not seen during training)
Benchmark inference latency (with/without adapter)
Measure VRAM footprint on target hardware
Test 10× actual peak load (verify stability)
Version control: store adapter + base model + training config
A/B test: compare merged vs runtime-loaded performance
Monitor: track inference latency, error rates, adapter-specific metrics
Fallback: plan behavior if adapter loading fails
Documentation: record training data, hyperparameters, expected quality

Multi-Adapter Serving Architecture

One of PEFT's greatest advantages: serve multiple task-specific adapters from a single frozen base model. This unlocks new architecture patterns.

Architecture Pattern: Single base model + multiple adapters enables task routing, adapter composition, and dynamic model updates without retraining or redeploying the base model.

Python — Multi-Adapter Serving with Request Routing
from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

class MultiAdapterServer:
    """Serves multiple task-specific adapters"""
    
    def __init__(self, base_model_name, adapter_configs):
        """
        base_model_name: HF model ID (e.g., 'meta-llama/Llama-2-7b-hf')
        adapter_configs: dict {task_name: adapter_path}
        """
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        
        # Load base model once
        self.model = AutoModelForCausalLM.from_pretrained(
            base_model_name,
            torch_dtype=torch.float16,
            device_map="auto"
        )
        self.tokenizer = AutoTokenizer.from_pretrained(base_model_name)
        
        # Load all adapters
        for task_name, adapter_path in adapter_configs.items():
            self.model = PeftModel.from_pretrained(
                self.model,
                adapter_path,
                adapter_name=task_name
            )
        
        self.active_adapter = None
    
    def detect_task(self, prompt):
        """Simple task detection (can be ML model in practice)"""
        keywords = {
            "support": ["help", "issue", "problem", "fix", "reset"],
            "coding": ["code", "python", "function", "write"],
            "analysis": ["analyze", "data", "statistics", "predict"],
        }
        
        prompt_lower = prompt.lower()
        for task, keywords_list in keywords.items():
            if any(kw in prompt_lower for kw in keywords_list):
                return task
        return "support"  # Default
    
    def generate(self, prompt, max_length=100, task=None):
        """Generate response with appropriate adapter"""
        
        # Auto-detect task if not specified
        if task is None:
            task = self.detect_task(prompt)
        
        # Switch adapter
        if task != self.active_adapter:
            self.model.set_adapter(task)
            self.active_adapter = task
        
        # Generate
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.device)
        with torch.no_grad():
            outputs = self.model.generate(
                **inputs,
                max_length=max_length,
                temperature=0.7,
                top_p=0.9
            )
        
        response = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
        return response

# Usage
server = MultiAdapterServer(
    "meta-llama/Llama-2-7b-hf",
    {
        "support": "adapters/support-lora",
        "coding": "adapters/code-lora",
        "analysis": "adapters/data-lora",
    }
)

# Requests automatically routed to correct adapter
response1 = server.generate("How do I reset my password?")  # → support adapter
response2 = server.generate("Write a Python merge sort")     # → coding adapter
response3 = server.generate("Analyze this sales data")       # → analysis adapter

Fine-Tuning from Scratch: Complete Example

End-to-end example from data loading to deployment.

Python — Complete Fine-Tuning Workflow
#!/usr/bin/env python3
"""
End-to-end LoRA fine-tuning on custom dataset
"""

from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from datasets import Dataset, load_dataset
import torch
import json

# ===== CONFIG =====
MODEL_NAME = "meta-llama/Llama-2-7b-hf"
OUTPUT_DIR = "./llama-custom-lora"
TRAINING_EPOCHS = 3
BATCH_SIZE = 8
LEARNING_RATE = 5e-4
LORA_RANK = 8

# ===== LOAD DATA (from JSON) =====
def load_custom_data(json_path):
    """Load training data from JSON"""
    with open(json_path) as f:
        data = json.load(f)
    return data

# Example JSON format:
# [
#   {"instruction": "...", "input": "...", "output": "..."},
#   ...
# ]

# ===== SETUP MODEL & TOKENIZER =====
print("Loading model...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# ===== APPLY LORA =====
print("Applying LoRA...")
lora_config = LoraConfig(
    r=LORA_RANK,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# ===== PREPARE DATA =====
print("Preparing data...")
data = load_custom_data("data.json")

def format_prompt(example):
    instruction = example.get("instruction", "")
    input_text = example.get("input", "")
    output = example.get("output", "")
    return f"Instruction: {instruction}\nInput: {input_text}\nOutput: {output}"

texts = [format_prompt(ex) for ex in data]

def tokenize_function(examples):
    outputs = tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,
        padding="max_length",
        return_tensors=None
    )
    outputs["labels"] = outputs["input_ids"].copy()
    return outputs

dataset = Dataset.from_dict({"text": texts})
dataset = dataset.map(tokenize_function, batched=True)

# Split: 90% train, 10% eval
split_dataset = dataset.train_test_split(test_size=0.1)

# ===== TRAINING =====
print("Starting training...")
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=TRAINING_EPOCHS,
    per_device_train_batch_size=BATCH_SIZE,
    per_device_eval_batch_size=BATCH_SIZE,
    warmup_steps=100,
    weight_decay=0.01,
    logging_steps=10,
    learning_rate=LEARNING_RATE,
    lr_scheduler_type="cosine",
    save_strategy="epoch",
    evaluation_strategy="epoch",
    seed=42,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=split_dataset["train"],
    eval_dataset=split_dataset["test"],
)

trainer.train()

# ===== SAVE & DEPLOY =====
print("Saving adapter...")
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

print("Merging for production...")
merged_model = model.merge_and_unload()
merged_model.save_pretrained(f"{OUTPUT_DIR}-merged")

print("✓ Complete! Deploy from: {OUTPUT_DIR}-merged

Rank Selection: Deep Analysis

Choosing the right rank is crucial for balancing quality and efficiency.

Empirical Rank Selection Rules

Baseline Rule: r=8 for any model (works 90% of time)

Task Complexity Scaling:
Simple (binary classification): r=4
Moderate (NLU tasks): r=8
Complex (reasoning, generation): r=16
Very complex (code, creative): r=32

Data-driven Selection:
Compute SVD of a few weight updates after 1 epoch
Measure variance explained at different ranks
Choose rank where 95% variance is captured

Grid Search Template:
For dataset_size ≤ 5k: test [4, 8, 16]
For dataset_size 5k-50k: test [8, 16, 32]
For dataset_size > 50k: test [16, 32, 64]

Key Insight: Rank selection has diminishing returns. Going from r=4 to r=8 might improve quality 3%. Going from r=32 to r=64 might improve 0.5%. Bigger gains come from dataset size, learning rate, and training duration.

Best Practices & Troubleshooting

Training Best Practices

Start Conservative

Begin with r=8, lr=5e-4, batch=8. Increase only if needed.

Monitor Continuously

Log loss every 10-20 steps. Use W&B or TensorBoard.

Save Checkpoints

Save adapter every epoch. Recover from crashes easily.

Validate Early

Check validation loss by epoch 1. Detect bad configs early.

Common Issues & Solutions

Loss not decreasing

LR too low (try 2-5×) OR rank too low (try r=16) OR bad data (verify labels). Check logs for NaN/Inf values.

Loss spikes/diverges

LR too high (reduce 50%). Try gradient clipping (max_norm=1.0). Reduce batch size.

Memory OOM errors

Reduce batch_size to 4 or 2. Enable gradient_checkpointing=True. Use QLoRA if available.

Poor final quality

Rank too low (increase to 16+). Dataset too small (collect more). Apply LoRA to more layers (Q,K,V,O,FFN).

Training loops forever

Set max_steps explicitly. Use early_stopping_patience. Monitor validation loss plateau.

Real-World Use Cases

Customer Support Chatbot

Fine-tune on company's historical support conversations. Deploy separate adapters for each product line (API, SaaS, Hardware). Route requests to task-specific adapters.

Code Generation for Proprietary Framework

Adapt CodeLLaMA on internal codebase and documentation. LoRA adapter learns coding style, libraries, and conventions specific to your stack.

Domain-Specific QA System

Medical QA: fine-tune on clinical literature. Legal QA: fine-tune on case law. Financial QA: fine-tune on regulatory documents. Single base model, multiple domain adapters.

Personalized AI Assistant

Users fine-tune on their own emails, documents, preferences. Deploy adapter locally on user's device. Privacy-preserving personalization.

A/B Testing Model Improvements

Cheap to train variants. Compare adapter A vs B vs C in production. Winner becomes deployed version. Faster iteration cycle.

Interview Questions & Answers

1. Explain the core mathematical insight behind LoRA.▼

LoRA exploits the observation that fine-tuning weight updates ΔW are intrinsically low-rank. Instead of updating the full m×n weight matrix W, we decompose the update as ΔW = BA where B ∈ ℝ^(m×r) and A ∈ ℝ^(r×n) with r ≪ min(m,n). This works because pre-training already learned powerful features; fine-tuning just recombines them for the specific task, which is a low-rank operation. Empirically, singular values of weight updates decay rapidly — rank 4-8 captures 95%+ variance.

2. Why does LoRA reduce training time compared to full fine-tuning?▼

Three reasons: (1) Fewer parameters to optimize (0.01-0.1% vs 100%) means smaller gradients and faster backprop. (2) Smaller optimizer states (Adam stores momentum + variance only for LoRA params). (3) Better numerical conditioning in lower-dimensional parameter space enables higher learning rates without divergence, reducing epochs needed. Combined: 5-10× speedup on single GPU.

3. Compare LoRA vs QLoRA. When would you use each?▼

LoRA: Base model in float16 (14GB for 7B), adapter in float32. Best quality, standard approach. Use when you have ≥24GB VRAM. QLoRA: Base model in 4-bit NF4 (3.5GB for 7B), adapter in float32. Trains 70B on 40GB A100s. Trade-off: 2-3% quality loss, 30% slower training. Use when GPU memory is tight (8-16GB). QLoRA not recommended if quality is critical and you have the hardware.

4. Explain the α/r scaling factor in LoRA. Why is it important?▼

LoRA update is scaled by α/r. Typically α=r. Without scaling, different ranks would have different update magnitudes, requiring different learning rates. With α=r, the effective learning rate is constant regardless of rank: the update (α/r)·BA·x ≈ same magnitude for any r. This enables hyperparameter transfer — you can change rank without retuning learning rate. Critical for reproducibility.

5. What are the limitations of LoRA?▼

(1) Low-rank assumption: For very different domains/tasks, higher rank might be needed, reducing efficiency gains. (2) Cannot modify architecture: Can't change vocab size, max position, or add/remove layers. (3) No inference speedup: LoRA reduces training/memory but not inference speed (unless merged). (4) Quality gap: 1-2% quality loss compared to full FT on some benchmarks. (5) Limited composition: Stacking multiple LoRAs can be unstable; adapters are better for composition.

6. How does inference speed change with LoRA?▼

If merged: identical to base model (LoRA weights are baked in). If loaded separately: ~1-2% overhead per token because you add BA·x computation in each forward pass. For 7B model, this is negligible (a few milliseconds). In practice, no noticeable latency difference for most applications. If inference latency is critical, merge before deployment.

7. How do you select the optimal rank for your task?▼

Start with r=8 (works for most cases). If validation loss plateaus early, increase to r=16. If still plateauing, try r=32. For rigorous selection: (1) Grid search [4, 8, 16, 32] on your validation set. (2) Plot loss vs rank — find the elbow where gains diminish. (3) Compute SVD of weight updates — choose rank where 95% variance is captured. In practice, r=8-16 is optimal for most tasks; rarely need higher.

8. How would you implement multi-task learning with PEFT adapters?▼

Train separate LoRA adapters for each task using the same base model. Load base model once, then load multiple adapters with different names. At inference, set_adapter(task_name) switches between tasks. For composition (e.g., "translate then summarize"), use Adapter layers instead of LoRA, as they support stacking and routing. This architecture enables efficient multi-task serving, easy A/B testing, and independent task updates.

Frequently Asked Questions (FAQ)

What is the difference between parameter-efficient fine-tuning and regular fine-tuning?▼

Regular fine-tuning updates all model parameters (billions). PEFT keeps most parameters frozen and trains only a small adapter (millions or less). PEFT: 60-90% less memory, 5-10× faster training, similar quality.

Can I use LoRA with any transformer model?▼

Yes, LoRA is architecture-agnostic. Works with any transformer: LLaMA, Mistral, Gemma, GPT, BERT, T5, etc. Just specify the weight matrix names you want to adapt. For GPT: target c_attn. For LLaMA: target q_proj, v_proj, etc. Use model.named_modules() to find layer names.

How long does LoRA training typically take?▼

1-3 hours on single modern GPU (A100/RTX4090) for 10k-100k examples. Scales roughly linearly with data size. Full FT takes 8-24 hours for same data. LoRA is 5-10× faster.

Do I need to use quantization with LoRA?▼

No, LoRA alone is usually sufficient. Use quantization (QLoRA) only if VRAM < 16GB. Trade-off: another 4× memory savings but 2-3% quality loss and 30% slower training. Decision tree: VRAM ≥ 24GB → LoRA. VRAM 16-24GB → LoRA maybe+checkpointing. VRAM < 16GB → QLoRA.

Can I merge a LoRA adapter and use it as a standalone model?▼

Yes! Use model.merge_and_unload() to bake adapter into base weights. Result is a standard HF model with no framework dependencies. Pro: universal compatibility. Con: lose modularity (can't swap adapters).

What is the typical size of a LoRA adapter?▼

For 7B model with rank=8: 50-100MB. Base model is 14GB. That is 140-280× smaller! For 70B with rank=8: 500-800MB. Makes distribution, versioning, and multi-adapter serving practical.

Does LoRA work well with instruction-following models like ChatGPT or Alpaca?▼

Absolutely! LoRA works great with any model. Just format your data in the same instruction-response style the model was trained on. Fine-tune with LoRA to specialize for your domain.

How do I handle vocabulary expansion with LoRA?▼

LoRA doesn't modify embedding matrices by default. Options: (1) Keep pre-trained tokenizer (simplest). (2) Add external expansion layer + LoRA adapters (complex). (3) Use full FT if vocabulary changes are essential.

Can I combine LoRA with other optimization techniques?▼

Yes, very effectively! LoRA + mixed precision (fp16) + gradient checkpointing work together. They target different bottlenecks: LoRA reduces parameters, fp16 reduces memory, checkpointing trades compute for memory. Combine all three for extreme efficiency.

What is the best way to version and track LoRA adapters in production?▼

Store adapters in Git (they're small!) or HF Hub. Include metadata files: base_model_version.txt, training_date, hyperparameters.json, performance_metrics.json. Use semantic versioning (v1.0, v1.1, v2.0). Track training runs with Weights & Biases. For A/B testing, save both adapter_a and adapter_b, deploy via feature flag.