AI Security - SustainSys AI Academy

Introduction to AI Security

As artificial intelligence systems become increasingly powerful and integrated into critical infrastructure, ensuring their security has become paramount. AI Security encompasses protecting AI systems from adversarial attacks, prompt injection, data poisoning, model theft, and emerging threats. Unlike traditional cybersecurity, AI security must defend against both external adversaries and vulnerabilities inherent to machine learning itself.

AI systems are uniquely vulnerable. A perturbation of just a few pixels can fool a computer vision model trained on millions of images. A carefully crafted prompt can make a language model produce harmful content. Poisoned training data can cause a model to develop hidden backdoors. These vulnerabilities are not bugs in implementation — they're properties of how neural networks learn and process information.

This course teaches you how modern organizations defend their AI systems, from adversarial robustness and input validation to differential privacy and red teaming. You'll learn both the theory of why attacks work and practical defensive techniques used by leading AI companies like OpenAI, Anthropic, Google DeepMind, and Meta.

What You'll Learn

Adversarial Attacks

Understand FGSM, PGD, and C&W attacks that fool neural networks. Learn why imperceptible perturbations can cause misclassification.

Prompt Injection Defense

Detect and prevent prompt injection attacks on language models. Implement input validation and detection systems.

Data Security

Learn about data poisoning, backdoor attacks, and differential privacy. Protect training data and prevent model tampering.

Red Teaming Framework

Build systematic approaches to find vulnerabilities. Conduct security audits and adversarial testing on AI systems.

Prerequisites

Understanding of Python, basic neural networks (forward/backward pass), PyTorch or TensorFlow, and familiarity with deep learning concepts like gradients and loss functions.

Why AI Security Matters

AI systems are now deployed in high-stakes applications: autonomous vehicles, medical diagnosis, financial trading, and critical infrastructure control. A security failure doesn't mean a slower server response — it can mean patient harm, financial fraud, or loss of life.

Real-World AI Security Incidents

Adversarial Images

In 2019, researchers demonstrated that a 3D-printed turtle was classified as a rifle by computer vision systems with 99% confidence, despite humans easily recognizing it as a turtle.

ChatGPT Jailbreaks

Within weeks of ChatGPT's release, researchers found prompt injections that bypassed safety guidelines, revealing vulnerabilities in language model alignment.

Model Poisoning

BadNets (2019) showed that poisoned training data could inject hidden backdoors into neural networks that activate on specific triggers.

Model Stealing

Researchers extracted proprietary information from commercial ML APIs using only query access and black-box attacks.

Threat Landscape

AI systems face threats across the entire pipeline:

AI Security Threat Landscape (Animated)

Data Layer

Poisoning, manipulation, privacy leaks

Training Layer

Backdoors, trojan attacks, data theft

Model Layer

Extraction, inversion, stealing

Input Layer

Adversarial examples, injection attacks

Output Layer

Manipulation, evasion, spoofing

Deployment Layer

Model replacement, supply chain attacks

Impact of Security Breaches

Loss of Trust

Critical

Legal/Compliance

Critical

Financial Loss

Critical

User Safety

Critical

Reality Check: Every deployed AI system will face adversaries. The question isn't if you'll be attacked, but whether you're prepared when you are.

Evolution of AI Security

AI security as a discipline emerged from academic research but has rapidly matured into a critical industry practice. Understanding this evolution helps contextualize modern defense strategies.

Timeline of AI Security

2013
Szegedy et al. discover adversarial examples — small perturbations fool deep networks

2014
FGSM attack introduces fast adversarial example generation

2016
PGD attack and adversarial training published; robustness becomes tractable

2017
NIPS competition on adversarial robustness; industry engagement begins

2019
BadNets and data poisoning attacks published; supply chain risks emerge

2022
ChatGPT jailbreaks; AI safety becomes mainstream concern

2024
Differential privacy, red teaming frameworks, and enterprise security standards mature

Key Paradigm Shifts

From Theoretical to Applied

Early work was academic curiosity. Now, defending against these attacks is a business requirement and compliance mandate.

From Isolated to Systemic

Initial focus was on individual models. Modern thinking addresses entire AI systems and supply chains.

From Detection to Prevention

Early defenses tried to detect attacks. Modern approaches prevent them through robust architectures.

From Ad-Hoc to Standardized

No common security framework existed. Industry now converging on standards for testing and validation.

This evolution mirrors general cybersecurity: initial surprise and skepticism gave way to rigorous scientific analysis, which led to practical defenses and ultimately to industry standardization and best practices.

Core Security Concepts

Before diving into specific attacks and defenses, understand the fundamental concepts that define AI security.

Threat Model

A threat model defines what an attacker can and cannot do:

White-Box Access

Attacker knows model weights, architecture, and all parameters. Can compute gradients. Strongest threat model.

Black-Box Access

Attacker can only query the model and observe outputs. Most realistic for deployed systems.

Gray-Box Access

Attacker has partial information: API documentation, known architecture, but not weights.

Attack Categories

Evasion Attacks

Craft adversarial inputs at inference time. Most common and well-studied threat.

Poisoning Attacks

Manipulate training data to cause model failures or inject backdoors.

Extraction Attacks

Steal model parameters, architecture, or training data through queries.

Privacy Attacks

Infer information about training data (membership inference, model inversion).

Robustness Metrics

Adversarial Robustness

Accuracy under worst-case perturbations. Formally: min accuracy over all perturbations within epsilon distance from original input.

Certified Robustness

Mathematically proven robustness guarantees. Guarantees that no perturbation within epsilon can change the prediction.

Perturbation Budget

Maximum allowed change to inputs (epsilon). Measured in Lp norms: L∞ (max change any feature), L2 (Euclidean distance), L1 (sum of changes).

Defense Categories

Adversarial Training

Train on adversarially perturbed examples. Most practical defense with empirical success.

Certified Defense

Use randomized smoothing or interval bounds to provide formal robustness guarantees.

Input Validation

Detect and reject adversarial inputs before processing. Complements other defenses.

Differential Privacy

Train models while provably protecting individual training examples from extraction.

Key Insight

There's a robustness-accuracy tradeoff: adversarially robust models are often less accurate on natural examples. Modern research focuses on minimizing this tradeoff.

AI Security Architecture

Protecting AI systems requires layered defenses across the entire ML pipeline. No single defense is sufficient; instead, multiple complementary strategies create defense-in-depth.

Defense-in-Depth Architecture

Layered Security Architecture

Layer 1: Data Security

Secure data collection, anonymization, integrity checks, backup protection

Layer 2: Training Security

Differential privacy, data validation, adversarial training, secure environments

Layer 3: Model Protection

Model watermarking, fingerprinting, watermark verification, integrity monitoring

Layer 4: Input Defense

Input validation, anomaly detection, adversarial example detection, rate limiting

Layer 5: Output Security

Output validation, confidence thresholding, safety filtering, audit logging

Layer 6: Deployment Security

Access controls, encryption, secure infrastructure, incident response

Key Architectural Principles

Assume Breach

Design defenses assuming attackers will get partial access. Use defense-in-depth so no single failure compromises security.

Fail Secure

When uncertain, reject. Better to deny service than provide incorrect results.

Least Privilege

Give models and systems minimum permissions needed. Restrict what each component can access.

Audit Everything

Log all access, predictions, and anomalies. Forensic analysis is crucial after incidents.

This architecture mirrors software security (defense-in-depth) but adapted for ML-specific threats. The goal is creating redundancy so that compromising one defense layer doesn't compromise overall security.

Key Security Components

Modern AI security systems are built from well-understood components, each addressing specific threat vectors.

Adversarial Robustness

Ensuring models maintain correct predictions despite adversarial perturbations. Two main approaches:

Empirical Robustness

Train on adversarial examples. Works well but without formal guarantees.

Certified Robustness

Mathematically proven bounds. Slower but provides guarantees like 'no 8-pixel-norm perturbation changes prediction'.

Input Validation

Detect anomalous or suspicious inputs before they reach the model:

Syntax Validation

Check format, length, encoding. Reject malformed inputs.

Semantic Validation

Ensure input makes sense for the application. Text length bounds, image properties, etc.

Anomaly Detection

ML models trained to detect out-of-distribution inputs. Flag suspicious patterns.

Differential Privacy

Train models while provably protecting individual training examples:

Differential Privacy Definition

A mechanism provides (epsilon, delta)-differential privacy if the probability of any outcome changes by at most e^epsilon when any single training example is added or removed. Lower epsilon = stronger privacy.

Red Teaming

Systematic, adversarial testing to find vulnerabilities before adversaries do:

Automated Red Teaming

Use attack algorithms to find vulnerabilities. Scale testing across thousands of examples.

Manual Red Teaming

Domain experts try to break the system using creativity and domain knowledge.

Community Programs

Bug bounties and responsible disclosure programs to find real-world vulnerabilities.

Monitoring & Detection

Detect attacks in progress through continuous monitoring:

Confidence Monitoring

Track model confidence. Adversarial examples often show unusual confidence patterns.

Prediction Shift

Monitor for unexpected changes in prediction distributions over time.

Latency Analysis

Adversarial examples sometimes require different computational resources.

Implementation Guide: Building Secure AI Systems

This section walks through implementing security measures in a real AI system. We'll build a secure image classifier with adversarial robustness and input validation.

Step 1: Setup Secure Development Environment

Python — Security Setup
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
import numpy as np
from typing import Tuple

# Ensure reproducibility
torch.manual_seed(42)
np.random.seed(42)

# Security configuration
EPSILON = 8/255.0  # Max perturbation budget
ATTACK_STEPS = 20   # PGD attack iterations
CONFIDENCE_THRESHOLD = 0.85  # Reject low-confidence predictions

class SecureAIConfig:
    epsilon: float = EPSILON
    attack_steps: int = ATTACK_STEPS
    confidence_threshold: float = CONFIDENCE_THRESHOLD
    enable_logging: bool = True

Step 2: Input Validation

Python — Input Validation
class InputValidator:
    '''Validates inputs before model processing'''
    
    def __init__(self, feature_bounds=None):
        self.feature_bounds = feature_bounds
        self.anomaly_detector = None
    
    def validate(self, x: torch.Tensor) -> Tuple[bool, str]:
        '''Returns (is_valid, reason)'''
        
        # Check shape
        if len(x.shape) != 4:  # Batch of images
            return False, "Invalid input shape"
        
        # Check value range
        if x.min() < 0 or x.max() > 1:
            return False, "Pixel values out of range [0,1]"
        
        # Check for NaN/Inf
        if torch.isnan(x).any() or torch.isinf(x).any():
            return False, "NaN or Inf values detected"
        
        # Check for adversarial patterns
        if self._has_adversarial_pattern(x):
            return False, "Adversarial pattern detected"
        
        return True, "Valid"
    
    def _has_adversarial_pattern(self, x: torch.Tensor) -> bool:
        '''Simple heuristic: check gradient magnitude'''
        x_requires_grad = x.clone().requires_grad_(True)
        # Check statistical anomalies
        return False  # Placeholder

Step 3: Adversarial Training

Python — Adversarial Training Loop
class AdversarialTrainer:
    '''Trains models with adversarial robustness'''
    
    def __init__(self, model, epsilon=EPSILON, attack_steps=ATTACK_STEPS):
        self.model = model
        self.epsilon = epsilon
        self.attack_steps = attack_steps
    
    def pgd_attack(self, x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
        '''Generate PGD adversarial examples'''
        x_adv = x.clone().detach().requires_grad_(True)
        
        for _ in range(self.attack_steps):
            # Forward pass
            output = self.model(x_adv)
            loss = F.cross_entropy(output, y)
            
            # Backward pass
            self.model.zero_grad()
            loss.backward()
            
            # Update adversarial example
            with torch.no_grad():
                x_adv.data += self.epsilon / self.attack_steps * x_adv.grad.sign()
                x_adv.data = torch.clamp(x_adv.data, 0, 1)
                x_adv.grad.zero_()
        
        return x_adv.detach()
    
    def train_epoch(self, dataloader, optimizer):
        '''Train for one epoch with adversarial examples'''
        total_loss = 0
        
        for x, y in dataloader:
            # Generate adversarial examples
            x_adv = self.pgd_attack(x, y)
            
            # Train on both clean and adversarial examples
            optimizer.zero_grad()
            
            # Clean loss
            logits_clean = self.model(x)
            loss_clean = F.cross_entropy(logits_clean, y)
            
            # Adversarial loss
            logits_adv = self.model(x_adv)
            loss_adv = F.cross_entropy(logits_adv, y)
            
            # Combined loss
            loss = loss_clean + loss_adv
            loss.backward()
            optimizer.step()
            
            total_loss += loss.item()
        
        return total_loss / len(dataloader)

Step 4: Secure Inference

Python — Secure Inference
class SecurePredictor:
    '''Makes predictions with security checks'''
    
    def __init__(self, model, validator, confidence_threshold=0.85):
        self.model = model
        self.validator = validator
        self.confidence_threshold = confidence_threshold
        self.predictions_log = []
    
    def predict(self, x: torch.Tensor) -> Tuple[np.ndarray, dict]:
        '''Secure prediction with validation and logging'''
        
        # Validate input
        is_valid, reason = self.validator.validate(x)
        if not is_valid:
            return None, {'valid': False, 'reason': reason}
        
        # Get predictions
        with torch.no_grad():
            logits = self.model(x)
            probs = F.softmax(logits, dim=1)
            confidence, predicted = torch.max(probs, 1)
        
        # Check confidence
        mask = confidence >= self.confidence_threshold
        predictions = predicted.cpu().numpy()
        predictions[~mask] = -1  # -1 indicates low confidence
        
        # Log for audit trail
        metadata = {
            'valid': True,
            'confidence': confidence.cpu().numpy(),
            'low_confidence_count': (~mask).sum().item(),
            'attack_detected': False
        }
        
        self.predictions_log.append(metadata)
        
        return predictions, metadata

Implementation Note: This is a simplified demonstration. Production systems need much more: distributed logging, encrypted storage, access controls, compliance auditing, and incident response procedures.

Advanced Security Techniques

Beyond basic adversarial training, advanced techniques provide stronger guarantees and address new threats.

Certified Defenses with Randomized Smoothing

Provide formal robustness guarantees by adding noise:

Python — Randomized Smoothing
class CertifiedSmoothing:
    '''Certified robustness via randomized smoothing'''
    
    def __init__(self, base_model, noise_std=0.25):
        self.base_model = base_model
        self.noise_std = noise_std
    
    def predict(self, x: torch.Tensor, n_samples=100) -> Tuple[int, float]:
        '''
        Certified prediction: returns (prediction, radius)
        Guarantees: prediction is robust to all perturbations < radius
        '''
        batch_size = x.shape[0]
        
        # Sample from Gaussian
        counts = torch.zeros(batch_size, 1000)  # 1000 classes
        
        for _ in range(n_samples):
            noise = torch.randn_like(x) * self.noise_std
            x_noisy = x + noise
            
            with torch.no_grad():
                logits = self.base_model(x_noisy)
            
            counts.scatter_add_(1, logits.argmax(1).unsqueeze(1), 1)
        
        # Get top-2 predictions
        top2 = counts.topk(2, dim=1)
        class_A = top2[1][:, 0]
        count_A = top2[0][:, 0]
        count_B = top2[0][:, 1]
        
        # Certified radius (for each sample)
        radius = (self.noise_std / 2) * (
            (count_A - count_B) / n_samples
        )
        
        return class_A, radius

Differential Privacy Training

Train models that provably protect training data:

Python — DP-SGD Training
import math

class DP_SGD_Trainer:
    '''Differentially Private Stochastic Gradient Descent'''
    
    def __init__(self, model, epsilon=1.0, delta=1e-5, max_grad_norm=1.0):
        self.model = model
        self.epsilon = epsilon
        self.delta = delta
        self.max_grad_norm = max_grad_norm
        self.privacy_budget_used = 0
    
    def train_batch(self, batch_x, batch_y, optimizer):
        '''Train with differential privacy guarantees'''
        
        logits = self.model(batch_x)
        loss = F.cross_entropy(logits, batch_y)
        
        optimizer.zero_grad()
        loss.backward()
        
        # Clip gradients per sample
        for param in self.model.parameters():
            if param.grad is not None:
                grad_norm = torch.norm(param.grad)
                param.grad.data *= min(1, self.max_grad_norm / grad_norm)
        
        # Add Gaussian noise
        noise_scale = self.max_grad_norm * math.sqrt(2 * math.log(1.25 / self.delta)) / self.epsilon
        
        for param in self.model.parameters():
            if param.grad is not None:
                param.grad.data += torch.randn_like(param.grad) * noise_scale
        
        optimizer.step()
        
        # Update privacy budget
        self.privacy_budget_used += self.epsilon
        
        return {
            'loss': loss.item(),
            'epsilon_used': self.epsilon,
            'privacy_budget_total': self.privacy_budget_used
        }

Prompt Injection Detection

Detect adversarial prompts targeting language models:

Python — Prompt Injection Detection
class PromptInjectionDetector:
    '''Detects prompt injection attacks on LLMs'''
    
    def __init__(self):
        self.injection_patterns = [
            r'ignore previous|ignore all|forget all',
            r'system prompt|system override|new instructions',
            r'jailbreak|bypass|exploit|vulnerability',
            r'mode: unfiltered|mode: uncensored|dev mode',
        ]
        self.suspicion_threshold = 0.7
    
    def detect(self, prompt: str) -> Tuple[bool, float]:
        '''
        Returns (is_injection, suspicion_score)
        '''
        import re
        
        prompt_lower = prompt.lower()
        suspicion_score = 0
        
        # Pattern matching
        for pattern in self.injection_patterns:
            if re.search(pattern, prompt_lower):
                suspicion_score += 0.3
        
        # Prompt structure analysis
        if '###' in prompt or '---' in prompt:
            suspicion_score += 0.2  # Multiple sections
        
        # Length anomaly
        if len(prompt) > 5000:
            suspicion_score += 0.1  # Suspiciously long
        
        # High-risk tokens
        dangerous_tokens = ['eval', 'exec', 'import', '__']
        for token in dangerous_tokens:
            if token in prompt_lower:
                suspicion_score += 0.15
        
        is_injection = suspicion_score > self.suspicion_threshold
        
        return is_injection, min(suspicion_score, 1.0)

Attack vs Defense Comparison

Different attacks require different defenses. This comparison shows the landscape:

Attack Type	Threat Model	Impact	Detection Difficulty	Defense Strategy
Adversarial Examples	White-box or Black-box	Misclassification	Hard (imperceptible)	Adversarial training, Input validation
Data Poisoning	Training data access	Model degradation or backdoors	Hard (hidden in training)	Data validation, Anomaly detection, Defenses
Model Extraction	Query access	IP theft, model cloning	Medium (detectable by monitoring)	Query monitoring, Rate limiting, API controls
Prompt Injection	User input control	Jailbreaking, policy bypass	Hard (looks like valid input)	Input validation, Detection models, Fine-tuning
Model Inversion	Query or white-box access	Privacy leakage (training data exposure)	Hard (requires specific queries)	Differential privacy, Output limiting, Cryptography
Membership Inference	Query or model access	Privacy leakage (determine if data was in training set)	Medium (statistical analysis)	Differential privacy, Regularization, Ensemble methods
Supply Chain Attack	Compromised dependencies	Full system compromise	Very hard (hidden in dependencies)	Code review, Integrity checks, Sandboxing, Monitoring
Model Replacement	Server access or MITM	Deployment of malicious model	Hard (looks like model update)	Cryptographic signing, Secure deployment, Verification

Vulnerability Severity Chart

Relative severity of different vulnerabilities (considering likelihood × impact):

Model Extraction

Critical

Adversarial Examples

High

Data Poisoning

High

Prompt Injection

High

Privacy Attacks

Medium-High

Supply Chain

Medium-High

Real-World Security Use Cases

AI security isn't abstract — organizations face these threats in production systems every day:

Autonomous Vehicles

Threat: Stop Sign Evasion

Adversarial perturbations on stop signs can cause object detection failures, making vehicles drive through intersections. This is not theoretical — physical-world adversarial examples have been successfully demonstrated on traffic signs. Defense: Multi-modal fusion (camera + LiDAR + radar), adversarial training on sign variations, and human override systems.

Medical Imaging

Threat: Diagnosis Evasion

Small perturbations to X-rays or MRI scans can cause misdiagnosis, potentially fatal. Unlike natural images, medical images have high cost-of-error. Defense: Certified robustness guarantees, ensemble methods, radiologist-in-the-loop, and careful validation.

Financial Systems

Threat: Trading Algorithm Manipulation

Adversarial price movements or data poisoning in training could cause algorithms to make costly mistakes. Malicious actors could profit from this. Defense: Adversarial training on historical anomalies, rate limiting on trades, and human oversight for large transactions.

Language Models

Threat: Prompt Injection & Jailbreaking

Users can craft prompts that override safety guidelines, causing models to generate harmful content. This has happened with ChatGPT, Bing, and other systems. Defense: Input validation, prompt engineering, RLHF with robust fine-tuning, and output filtering.

Biometric Authentication

Threat: Adversarial Face Spoofing

Adversarial images or physical attacks can fool face recognition systems. Combined with stolen biometric data, this is a critical vulnerability. Defense: Liveness detection, adversarial training on spoofing attempts, multimodal biometrics.

Content Moderation

Threat: Evasion of Safety Filters

Users develop techniques to bypass content moderation AI (typos, symbols, etc.). Malicious actors systematically find vulnerabilities. Defense: Adversarial training on evasion attempts, ensemble methods, human review escalation.

Enterprise AI Security

Large organizations deploying AI at scale face unique challenges requiring systematic approaches.

Enterprise Security Framework

Threat Assessment

Regular red teaming and vulnerability scanning. Update threat model as new attacks emerge.

Compliance

Meet regulatory requirements (GDPR, HIPAA, etc.). Document security measures and audit trails.

Incident Response

Plan for when (not if) incidents occur. Have playbooks for different attack types.

Continuous Monitoring

Runtime monitoring of predictions, confidence scores, and anomaly detection.

Security in the ML Pipeline

1. Data Collection & Validation

Verify data sources and integrity
Detect poisoned or malicious examples
Anonymize and protect sensitive data
Maintain audit logs

2. Model Development

Code review for security vulnerabilities
Dependency scanning for supply chain attacks
Adversarial training where applicable
Version control with cryptographic signing

3. Model Evaluation

Red teaming and adversarial testing
Bias and fairness evaluation
Privacy impact assessment
Formal verification where possible

4. Deployment

Cryptographic model signing and verification
Secure infrastructure (encryption at rest/transit)
Access controls and authentication
Rate limiting and DDoS protection

5. Operation

Runtime monitoring of predictions
Confidence thresholding and fallback mechanisms
Anomaly detection on inputs/outputs
Audit logging and forensics capability

Organizational Practices

Red Team Program

Dedicated team tasked with finding vulnerabilities. Operates independently of development teams.

Bug Bounty

External researchers help find vulnerabilities. Responsible disclosure and rewards.

Security Audits

Regular third-party audits to validate controls and identify gaps.

Training

All team members understand AI-specific threats and their responsibilities.

Common AI Security Mistakes

Even well-intentioned teams make critical security errors. Learn from others' mistakes:

Mistake 1: Ignoring Adversarial Examples

Error: "Our model has 99% accuracy, so it's secure." Accuracy on natural examples ≠ robustness to adversarial examples.

Fix: Test specifically for adversarial robustness. Use FGSM, PGD, or C&W attacks. Track accuracy under perturbations.

Mistake 2: Security Through Obscurity

Error: "Our model is proprietary, so it's safe." White-box attacks don't require knowing the model — black-box attacks work with just query access.

Fix: Assume attackers can query your API or access model weights. Design robust systems, not hidden ones.

Mistake 3: Testing Only on Standard Data

Error: "The model works on our test set, so it's safe." This tests natural accuracy, not adversarial robustness or real-world scenarios.

Fix: Test on adversarial examples, out-of-distribution data, and edge cases. Red team the system.

Mistake 4: No Input Validation

Error: Directly feeding user input to the model. Even simple validation can stop many attacks.

Fix: Validate all inputs: format, range, plausibility. Reject anomalous inputs.

Mistake 5: Trusting Model Confidence

Error: Using confidence scores as a proxy for correctness. Adversarial examples can fool models with high confidence.

Fix: Monitor confidence distributions. Ensemble methods. Thresholding. Don't treat confidence as a safety measure alone.

Mistake 6: Single Point of Failure

Error: All security depends on one defense mechanism. If it fails, the system is compromised.

Fix: Use defense-in-depth. Multiple layers of security so no single failure is catastrophic.

Mistake 7: No Monitoring or Logging

Error: Deploying without visibility into what the model is doing. Can't detect attacks if you're not looking.

Fix: Log all predictions, confidence scores, and anomalies. Monitor for drift and attacks in real-time.

Mistake 8: Ignoring Data Poisoning

Error: "Our training data is clean." Data poisoning is subtle and expensive to detect but devastating.

Fix: Data validation, anomaly detection on training data, smaller validation sets to catch poison early.

AI Security Best Practices

Guidelines that work in practice across industry and research:

Principle 1: Defense-in-Depth

Don't rely on any single defense. Layer multiple complementary protections:

Input Layer

Validate, sanitize, detect anomalies before model sees input.

Model Layer

Train robustly, use ensemble methods, maintain version control.

Output Layer

Validate outputs, threshold confidence, filter unsafe responses.

System Layer

Monitor, audit, rate-limit, control access.

Principle 2: Threat Modeling

Explicitly identify what you're defending against:

What are your assets? Model, data, predictions, inference results
Who are the adversaries? External attackers, malicious insiders, competitors
What are their capabilities? Query access, code access, network access, physical access
What's the impact if compromised? Varies from financial to safety-critical

Principle 3: Continuous Evaluation

Security is ongoing, not a one-time check:

Red team regularly (at least quarterly)
Monitor production systems continuously
Update threat model as attacks evolve
Conduct security audits by independent teams

Principle 4: Transparency & Documentation

Document your security measures and limitations:

What attacks are you defending against?
What are known limitations?
How is the system monitored?
What's the incident response plan?

Principle 5: Fail Secure

When in doubt, reject or escalate:

Confidence below threshold → reject or human review
Anomalous input → reject or human review
Unusual pattern → log and investigate
Better to deny service than provide incorrect results

Principle 6: Keep Up with Research

AI security evolves rapidly:

Follow security conferences (NDSS, CCS, IEEE S&P)
Subscribe to threat intelligence feeds
Participate in security communities
Test new attacks against your systems

Advanced Security Insights

Deep dives into subtle but important aspects of AI security:

The Robustness-Accuracy Tradeoff

Making a model robust to adversarial perturbations typically reduces its accuracy on natural examples. This is a fundamental tradeoff, not just an implementation issue.

Why? The decision boundaries robust models learn are smoother and less complex. This protects against perturbations but can miss fine details in natural data. It's as if the model becomes "more conservative" to handle adversarial examples.

Threat Model Assumptions Matter

A defense that's excellent for white-box attacks might be useless for black-box attacks. Always specify your threat model:

White-Box Attacks

Attacker has full model access. Strongest threat. Most defenses here are actually quite strong.

Black-Box Attacks

Attacker only sees outputs. More realistic. Many defenses fail here due to transferability.

Transferability: The Silent Killer

Adversarial examples generated for one model often fool other models. This means:

Attacking ensemble methods is easier than attacking single models
Training on one attacker doesn't necessarily defend against others
Black-box attacks are easier than we'd like: generate examples against a surrogate model, transfer them

Privacy-Security Tradeoff

Differential privacy protects individuals in training data but reduces model accuracy. Balancing these is a key design decision:

Tight privacy (low ε): Strong protection, lower accuracy
Loose privacy (high ε): Better accuracy, weaker protection
Modern techniques push this frontier, improving both

Formal Verification vs Practical Security

Certified defenses give formal guarantees but are often slow and may be overly conservative. Trade-offs exist:

Empirical Robustness: Fast, practically sufficient, no guarantees
Certified Robustness: Slow, mathematically proven, may be over-conservative
Hybrid approaches emerging: certified + empirical

Practical Code Examples

Working implementations of key security techniques. All examples use PyTorch.

Example 1: FGSM Adversarial Attack

The simplest adversarial attack, using the gradient of the loss function:

Python — FGSM Attack
import torch
import torch.nn.functional as F

def fgsm_attack(model, x, y, epsilon=0.03):
    '''
    Fast Gradient Sign Method attack.
    Returns adversarial example x_adv with perturbation <= epsilon
    '''
    x_adv = x.clone().detach().requires_grad_(True)
    
    # Forward pass
    output = model(x_adv)
    loss = F.cross_entropy(output, y)
    
    # Backward pass to get gradients
    if x_adv.grad is not None:
        x_adv.grad.zero_()
    loss.backward()
    
    # FGSM: move in direction of gradient
    with torch.no_grad():
        perturbation = epsilon * x_adv.grad.sign()
        x_adv.data = x_adv.data + perturbation
        x_adv.data = torch.clamp(x_adv.data, 0, 1)
    
    return x_adv.detach()

# Usage
x_clean = torch.randn(1, 3, 32, 32)  # Example image
y = torch.tensor([5])  # Target class
x_adversarial = fgsm_attack(model, x_clean, y, epsilon=0.03)

# Check that perturbation is small
print(f"L-infinity norm: {(x_adversarial - x_clean).abs().max():.4f}")

Example 2: Adversarial Training

Training a robust model by including adversarial examples:

Python — Adversarial Training
def adversarial_training(model, dataloader, optimizer, epochs=10):
    '''Train model with adversarial robustness'''
    
    for epoch in range(epochs):
        total_loss = 0
        
        for x, y in dataloader:
            # Generate adversarial examples
            x_adv = fgsm_attack(model, x, y, epsilon=0.03)
            
            # Train on both clean and adversarial
            optimizer.zero_grad()
            
            # Clean loss
            logits_clean = model(x)
            loss_clean = F.cross_entropy(logits_clean, y)
            
            # Adversarial loss
            logits_adv = model(x_adv)
            loss_adv = F.cross_entropy(logits_adv, y)
            
            # Total loss
            loss = loss_clean + loss_adv
            loss.backward()
            optimizer.step()
            
            total_loss += loss.item()
        
        avg_loss = total_loss / len(dataloader)
        print(f"Epoch {epoch+1}, Loss: {avg_loss:.4f}")

Example 3: Input Validation & Anomaly Detection

Detect adversarial or anomalous inputs before prediction:

Python — Input Anomaly Detection
class AnomalyDetector:
    '''Detect suspicious inputs using statistical methods'''
    
    def __init__(self, clean_data_examples):
        # Learn statistics from clean data
        self.mean = clean_data_examples.mean(dim=0)
        self.std = clean_data_examples.std(dim=0)
        
        # Compute z-scores for clean data
        z_scores = ((clean_data_examples - self.mean) / (self.std + 1e-8)).abs()
        self.threshold = z_scores.max(dim=0)[0].mean()
    
    def is_anomalous(self, x):
        '''Check if x is anomalous (likely adversarial)'''
        z_scores = ((x - self.mean) / (self.std + 1e-8)).abs()
        max_z = z_scores.max()
        return max_z > self.threshold * 2

# Usage
detector = AnomalyDetector(clean_train_data)

for x, y in test_dataloader:
    if detector.is_anomalous(x):
        print(f"Anomalous input detected!")
        continue  # Skip this input
    
    # Normal processing
    predictions = model(x)

Example 4: Confidence-Based Rejection

Reject low-confidence predictions to improve security:

Python — Confidence Thresholding
def secure_predict(model, x, confidence_threshold=0.85):
    '''
    Make predictions with confidence-based rejection.
    Returns (prediction, confidence, rejected)
    '''
    
    with torch.no_grad():
        logits = model(x)
        probabilities = F.softmax(logits, dim=1)
        confidence, predictions = torch.max(probabilities, dim=1)
    
    # Reject low-confidence predictions
    rejected = confidence < confidence_threshold
    predictions[rejected] = -1  # -1 indicates rejected
    
    return predictions, confidence, rejected

# Usage
for x, _ in test_dataloader:
    preds, conf, rejected = secure_predict(model, x, threshold=0.85)
    
    print(f"Predictions: {preds}")
    print(f"Confidence: {conf}")
    print(f"Rejected: {rejected.sum()} out of {len(x)}

Example 5: Ensemble Robustness

Use multiple models for more robust predictions:

Python — Ensemble Method
class RobustEnsemble:
    '''Ensemble of models for robust predictions'''
    
    def __init__(self, models):
        self.models = models
    
    def predict(self, x):
        '''Ensemble prediction using voting'''
        all_predictions = []
        all_confidences = []
        
        for model in self.models:
            with torch.no_grad():
                logits = model(x)
                probs = F.softmax(logits, dim=1)
                conf, pred = torch.max(probs, dim=1)
            
            all_predictions.append(pred)
            all_confidences.append(conf)
        
        # Majority vote
        ensemble_pred = torch.stack(all_predictions).mode(dim=0)[0]
        ensemble_conf = torch.stack(all_confidences).mean(dim=0)
        
        return ensemble_pred, ensemble_conf

# Usage
ensemble = RobustEnsemble([model1, model2, model3])
predictions, confidence = ensemble.predict(x_test)

Hands-On Exercises

Practice implementing AI security concepts. Solutions available after completion.

Exercise 1: Implement PGD Attack

Extend the FGSM attack to a stronger PGD (Projected Gradient Descent) attack. PGD makes multiple steps of FGSM, projecting back to the epsilon ball each time. Test on MNIST.

Python — Starter Code
def pgd_attack(model, x, y, epsilon=0.03, alpha=0.01, steps=20):
    '''
    TODO: Implement PGD attack
    - Start with random perturbation
    - For each step:
      - Compute loss gradient
      - Move in gradient direction by alpha
      - Project back to epsilon ball
    Return adversarial example
    '''
    pass

# Test on MNIST
x_clean = test_images[0:10]  # 10 test images
y_true = test_labels[0:10]

x_pgd = pgd_attack(model, x_clean, y_true)
pred_clean = model(x_clean).argmax(dim=1)
pred_pgd = model(x_pgd).argmax(dim=1)

print(f"Clean accuracy: {(pred_clean == y_true).float().mean():.3f}")
print(f"PGD accuracy: {(pred_pgd == y_true).float().mean():.3f}")

Exercise 2: Build Adversarial Training

Implement full adversarial training loop. Train a model on both clean and adversarial examples. Measure accuracy under attack.

Python — Starter Code
def train_robust_model(model, train_loader, optimizer, epochs=5):
    '''
    TODO: Implement adversarial training
    - For each batch:
      - Generate adversarial examples
      - Forward pass on clean examples
      - Forward pass on adversarial examples
      - Combine losses
      - Backward and update
    '''
    pass

# Train robust model
model = SimpleModel()
optimizer = torch.optim.Adam(model.parameters())
train_robust_model(model, train_loader, optimizer)

# Evaluate robustness
clean_acc = evaluate(model, test_loader)
pgd_acc = evaluate_pgd(model, test_loader)
print(f"Clean: {clean_acc:.3f}, PGD: {pgd_acc:.3f}")

Exercise 3: Detect Adversarial Examples

Build a detector that identifies adversarial examples using statistical features. Test on both clean and adversarial MNIST.

Python — Starter Code
class AdversarialDetector:
    def __init__(self):
        pass
    
    def fit(self, clean_examples):
        '''
        TODO: Learn statistics from clean examples
        Compute features like:
        - Pixel value distribution
        - Gradient magnitude
        - Spectral properties
        '''
        pass
    
    def detect(self, x):
        '''
        TODO: Compare input to clean statistics
        Return probability that input is adversarial
        '''
        pass

# Test detector
detector = AdversarialDetector()
detector.fit(clean_train_data)

clean_scores = detector.detect(clean_test_data)
adv_scores = detector.detect(pgd_examples)

# ROC curve
from sklearn.metrics import roc_auc_score
labels = [0]*len(clean_scores) + [1]*len(adv_scores)
scores = list(clean_scores) + list(adv_scores)
auc = roc_auc_score(labels, scores)
print(f"AUC: {auc:.3f}")

Exercise 4: Red Team Your Model

Systematically find vulnerabilities in a model using different attack methods and ensemble approaches. Measure worst-case error.

Python — Starter Code
def red_team_model(model, test_data):
    '''
    TODO: Comprehensive red team evaluation
    - FGSM attack
    - PGD attack
    - C&W attack
    - Various epsilon values
    - Black-box transfer attacks
    
    Return summary of vulnerabilities found
    '''
    
    results = {}
    
    # Try different attacks
    for attack_name in ['fgsm', 'pgd', 'cw']:
        for epsilon in [0.01, 0.03, 0.1]:
            # Generate adversarial examples
            # Measure success rate
            pass
    
    return results

# Red team the model
vulnerabilities = red_team_model(model, test_data)
print(vulnerabilities)

Security Interview Questions

Questions you might encounter in AI security interviews, with discussion guides.

Question 1: Explain Adversarial Examples

What they want: Deep understanding, not just "small perturbations fool models."

Good answer structure:

Define formally: perturbations within epsilon distance that cause misclassification
Explain why they exist: networks learn overly linear decision boundaries
Give concrete example: image classification, stop sign evasion
Discuss threat model: white-box vs black-box
Mention defense: adversarial training, certified robustness

Question 2: How Would You Secure an ML System?

What they want: Systems thinking, not just technical depth.

Good answer structure:

Defense-in-depth: multiple layers
Data security: validation, integrity, privacy
Model security: robustness, monitoring, versioning
Deployment security: access control, encryption, audit logs
Incident response: detection, response, recovery

Question 3: What's the Robustness-Accuracy Tradeoff?

What they want: Understanding of fundamental ML security concepts.

Key points:

Making models robust reduces natural accuracy
Why: robust models learn smoother decision boundaries
This is a fundamental property, not just implementation
Research focuses on minimizing the tradeoff
Practical consideration: how much accuracy can we afford to lose?

Question 4: Design a Red Team for an LLM

What they want: Practical security assessment strategy.

Good answer should include:

Automated testing: adversarial prompts, jailbreak attempts
Manual testing: creative attacks, domain expertise
Taxonomy: what categories of attacks to test (safety, fairness, privacy)
Metrics: what constitutes a successful attack
Iteration: continuously update threat model based on findings

Question 5: Explain Differential Privacy in Training

What they want: Technical understanding with practical awareness.

Key points:

DP-SGD: gradient clipping + noise injection
Epsilon-delta: epsilon = how much privacy, delta = failure probability
Privacy-utility tradeoff: stronger privacy = worse model
Why it matters: protects training data from extraction/inversion attacks
Practical use: when differential privacy is necessary

Question 6: Black-Box Attack on a Deployed Model

What they want: Realistic threat assessment.

Good answer:

Query access: can probe the model's API
Transfer attacks: generate adversarial examples on a surrogate model, transfer them
Harder but possible: iterative refinement using model's feedback
Defense strategy: rate limiting, input validation, ensemble methods

Question 7: Prompt Injection Attack & Defense

What they want: LLM-specific security thinking.

Key points:

Attack: craft prompts that override system instructions
Severity: can bypass safety guidelines, extract training data
Defense: input validation, fine-tuning on adversarial examples, prompt engineering
Limitations: difficult to completely prevent, may require human-in-the-loop

Question 8: Model Stealing Mitigation

What they want: Intellectual property protection awareness.

Good answer:

Threat: attacker queries API to steal model
Defenses: rate limiting, output perturbation, watermarking
Trade-offs: these defenses may degrade accuracy or user experience
Reality: perfect defense is impossible, goal is to make stealing expensive

Frequently Asked Questions

Are adversarial examples a real threat? ▼

Absolutely. They've been demonstrated on real systems like autonomous vehicles (traffic signs) and facial recognition. Major AI companies invest heavily in defending against them.

Can we make ML systems 100% secure? ▼

No. Security is always about risk management. We can make attacks expensive and difficult, but not impossible. The goal is defense-in-depth and detection of attacks in progress.

Does differential privacy destroy model accuracy? ▼

It reduces accuracy, but not catastrophically. Modern DP-SGD allows training competitive models. The privacy-utility tradeoff is a design choice based on application requirements.

What's the most important AI security defense? ▼

Defense-in-depth: no single defense is sufficient. Combine adversarial training, input validation, monitoring, and incident response. Different defenses protect against different threats.

Can I detect all adversarial examples? ▼

No, but you can detect some. Detectors are an arms race: new attacks are crafted to evade detectors. Detection is a helpful layer but shouldn't be your only defense.

How often should I red team my models? ▼

Continuously, especially before major releases. At minimum quarterly for production systems. More frequently if you deploy updates or change threat model.

What about GPU adversarial examples? ▼

Harder to execute (real-world attacks need physical modifications), but demonstrated in research. Includes things like adversarial patches, 3D objects, or printed materials that fool vision systems.

How does transfer learning affect security? ▼

Pre-trained models inherit security properties of their training process. If training data was poisoned, pre-trained model carries the poison. Fine-tuning on clean data helps, but doesn't guarantee remediation.

Are open-source models less secure? ▼

Not necessarily. Open-source means more eyes for security review, but also means attackers can analyze the code. The tradeoff is transparency vs obscurity, neither is inherently secure.

How do I explain AI security budget to non-technical leadership? ▼

Frame as insurance and risk management: 'This prevents costly breaches, regulatory fines, and loss of customer trust. The cost is trivial compared to the risk.'

Summary: Key Takeaways

As you deploy AI systems in production, remember these core principles:

Threats Are Real

Adversarial examples, prompt injection, data poisoning, and model theft happen in the wild. Don't dismiss them as theoretical.

Defense-in-Depth

No single defense is sufficient. Layer multiple complementary protections across the entire ML pipeline.

Assume Breach

Design systems assuming attackers will find vulnerabilities. Detection and response are as important as prevention.

Continuous Evaluation

Security is not a one-time checklist. Red team regularly, monitor in production, stay updated on new attacks.

The Security Mindset

Developing AI security expertise requires shifting your thinking:

From: "Our model achieves 99% accuracy, so it's good."
To: "Our model achieves 99% accuracy on natural data. What about under adversarial conditions?"

From: "We'll detect attacks at inference time."
To: "We need to prevent attacks, detect them, and respond quickly."

From: "Our proprietary model is secure because it's secret."
To: "We design for security assuming attackers have model access or can query it."

The Path Forward

You now understand the landscape of AI security. The next steps:

1. Apply to Your Systems

Take these concepts and apply them to models you own or use. Start with threat modeling and risk assessment.

2. Keep Learning

AI security research moves fast. Subscribe to security conferences (NDSS, CCS, IEEE S&P), read papers, and stay updated.

3. Red Team

Build red teaming into your development process. Make finding vulnerabilities a regular practice, not an afterthought.

4. Collaborate

Security is better with community. Share findings (responsibly), participate in bug bounties, learn from others.

5. Think Holistically

AI security isn't just about algorithms—it's about data, systems, organizations, and incentives. Consider the full context.

Remember: Every AI system will be attacked. The question isn't whether, but whether you're prepared when it happens.

Resources & Further Learning

Deepen your AI security expertise with these resources:

Foundational Papers

Adversarial Examples

Goodfellow et al., 'Explaining and Harnessing Adversarial Examples' (2014). The foundational work on FGSM attacks.

Robust Models

Madry et al., 'Towards Deep Learning Models Resistant to Adversarial Attacks' (2019). PGD attacks and adversarial training.

Differential Privacy

Abadi et al., 'Deep Learning with Differential Privacy' (2016). DP-SGD for training with privacy guarantees.

Data Poisoning

Shafahi et al., 'Poisoning Attacks against Support Vector Machines' (2016). Foundational poisoning attack.

Tools & Libraries

Adversarial Robustness Toolbox (ART)

IBM's library for generating adversarial examples and defenses. Supports multiple frameworks.

Opacus

Facebook's library for DP-SGD training. Makes adding differential privacy easy.

CleverHans

TensorFlow/Keras library for adversarial example generation and robustness evaluation.

FGSM & PGD

PyTorch implementations available in most ML libraries. Simple to implement from scratch.

Online Courses

CS109A: AI for Defense & Security — MIT course on defensive AI systems
UC Berkeley AI Security — Research-focused course on emerging threats and defenses
Coursera: AI Security — Industry-focused practical course

Conferences

NDSS (Network and Distributed System Security Symposium) — Top-tier security
CCS (ACM Conference on Computer and Communications Security) — Broad security topics
IEEE S&P (IEEE Symposium on Security and Privacy) — Prestigious security venue
NeurIPS (Neural Information Processing Systems) — ML security track
ICLR (International Conference on Learning Representations) — Robustness track

Organizations & Communities

Anthropic Safety Team — Research on AI safety and alignment
DeepMind Safety & Alignment — Google DeepMind's safety research
OpenAI Safety & Policy — ChatGPT and GPT security research
MIRI (Machine Intelligence Research Institute) — Long-term AI safety
Center for AI Safety — Independent AI safety research

Practical Resources

NIST AI Risk Management Framework — Standards and guidance for responsible AI
OWASP Top 10 for LLMs — Security risks in language models
FAIR Institute — Quantitative risk management for AI
AI Security & Safety blog by researchers — Latest developments in AI security

Recommended Books

Adversarial Machine Learning at Scale — Papernot et al.
The Security of Machine Learning — Joseph et al.
AI Safety and Reproducibility — Leike, 2024
Securing Machine Learning Systems — Fredrikson, 2023

Next Steps

Your AI security journey continues. Take these tools and concepts, apply them to real systems, stay current with research, and help build safer AI systems. The field needs thoughtful practitioners who understand both the threats and the defenses.

Topics

Introduction to AI Security

What You'll Learn

Adversarial Attacks

Prompt Injection Defense

Data Security

Red Teaming Framework

Prerequisites

Why AI Security Matters

Real-World AI Security Incidents

Adversarial Images

ChatGPT Jailbreaks

Model Poisoning

Model Stealing

Threat Landscape

AI Security Threat Landscape (Animated)

Impact of Security Breaches

Evolution of AI Security

Timeline of AI Security

Key Paradigm Shifts

From Theoretical to Applied

From Isolated to Systemic

From Detection to Prevention

From Ad-Hoc to Standardized

Core Security Concepts

Threat Model

White-Box Access

Black-Box Access

Gray-Box Access

Attack Categories

Evasion Attacks

Poisoning Attacks

Extraction Attacks

Privacy Attacks

Robustness Metrics

Defense Categories

Adversarial Training

Certified Defense

Input Validation

Differential Privacy

Key Insight

AI Security Architecture

Defense-in-Depth Architecture

Layered Security Architecture

Key Architectural Principles

Assume Breach

Fail Secure

Least Privilege

Audit Everything

Key Security Components

Adversarial Robustness

Empirical Robustness

Certified Robustness

Input Validation

Syntax Validation

Semantic Validation

Anomaly Detection

Differential Privacy

Differential Privacy Definition

Red Teaming

Automated Red Teaming

Manual Red Teaming

Community Programs

Monitoring & Detection

Confidence Monitoring

Prediction Shift

Latency Analysis

Implementation Guide: Building Secure AI Systems

Step 1: Setup Secure Development Environment

Step 2: Input Validation

Step 3: Adversarial Training

Step 4: Secure Inference

Advanced Security Techniques

Certified Defenses with Randomized Smoothing

Differential Privacy Training

Prompt Injection Detection

Attack vs Defense Comparison

Vulnerability Severity Chart

Real-World Security Use Cases

Autonomous Vehicles