MLOps - SustainSys AI Academy

Introduction to MLOps

MLOps (Machine Learning Operations) bridges the gap between ML development and production systems. It combines the principles of DevOps with machine learning, focusing on operationalizing models, automating workflows, and ensuring systems are reliable, scalable, and maintainable.

In 2024, the industry consensus is clear: building a model is only 5% of the work. The remaining 95% is infrastructure, monitoring, retraining, and maintenance. This section will guide you through every critical aspect of MLOps, from experiment tracking through production deployment and monitoring.

What This Page Covers

You'll learn about:

Experiment Tracking: MLflow, Weights & Biases, Neptune
Containerization: Docker for reproducible ML environments
Model Serving: FastAPI, TorchServe, KServe
CI/CD Pipelines: GitHub Actions, GitLab CI, automating ML workflows
Kubernetes Deployment: Orchestrating containerized models at scale
Monitoring & Observability: Production model behavior tracking
Data Drift Detection: Using Evidently, Great Expectations

Prerequisites

This guide assumes you understand Python, machine learning basics, and version control (Git). Familiarity with Docker and cloud platforms is helpful but not required.

Why MLOps Matters

The ML Project Lifecycle

A typical ML project spans multiple phases, each with distinct operational challenges:

Development Phase

Data scientists experiment with models, hyperparameters, and features. Tools: Jupyter, MLflow, Git. Challenge: dozens of experiments with no versioning.

Validation Phase

Models are tested against holdout data, cross-validation, and edge cases. Challenge: reproducing results across machines.

Deployment Phase

Models move to production for real-world predictions. Challenge: ensuring performance, latency, and reliability.

Monitoring Phase

Models perform inference on live data. Challenge: data distribution shifts, model drift, silent failures.

Maintenance Phase

Models are retrained, versioned, and rolled back as needed. Challenge: automating the entire pipeline.

Key MLOps Benefits

Reproducibility

Every experiment is tracked with code, data, and hyperparameters. Reproduce any result months later.

Speed to Market

Automated pipelines deploy models from code to production in hours, not weeks.

Risk Reduction

Automated testing, canary deployments, and monitoring catch failures before users see them.

Team Collaboration

Data scientists, engineers, and ops work with shared tools, reducing handoffs and friction.

Cost Efficiency

Efficient resource allocation, model optimization, and automated scaling reduce cloud costs.

The MLOps Maturity Journey

Most organizations evolve through predictable stages:

Ad Hoc: Manual experiments, no versioning, notebooks in email

Level 1

Organized: Git repos, experiment tracking, but manual deployments

Level 2

Operationalized: Automated CI/CD, model registry, monitoring alerts

Level 3

Advanced: Automated retraining, A/B testing, feature stores

Level 4

World-Class: End-to-end automation, multi-model ensembles, zero-downtime updates

Level 5

Core MLOps Concepts

Model Versioning

Unlike code versioning (git), models require tracking input features, training data, hyperparameters, and metrics. A model version is uniquely identified by:

Code: Exact training script (git commit SHA)
Data: Training/test data fingerprint or git-like hash
Parameters: Hyperparameters (learning rate, layers, etc.)
Artifacts: Serialized model weights (pickle, ONNX, SavedModel)
Metrics: Evaluation metrics (accuracy, F1, AUC, etc.)

Model Registry Pattern

A central repository of production models with metadata. Examples: MLflow Model Registry, HuggingFace Hub, cloud-provided registries. Tracks which version is in production, staging, and development.

Experiment Tracking

Record every experiment's parameters, metrics, and artifacts. Benefits:

Compare models side-by-side
Reproduce previous results
Identify which hyperparameters matter most
Share results with the team
Audit which model made a prediction

Feature Stores

Centralized repositories of processed features with versioning, lineage, and monitoring. Problems solved:

Training-serving skew: Same features in training and production
Duplicate work: Features computed once, reused everywhere
Data lineage: Track which raw data feeds which features

Production Feature Store Examples

Tecton, Feast, Databricks Feature Store. Map raw data to ML-ready features automatically.

Model Serving Patterns

How models go from disk to serving predictions in production:

Batch Serving: Process entire datasets at once (daily/hourly). High throughput, high latency.
Real-time Serving: HTTP API responds to individual requests. Low latency, resource-intensive.
Embedded: Model runs inside the application (mobile, edge). No network latency.
Streaming: Process continuous event streams. Complex orchestration.

Data Drift vs Model Drift

Data Drift: The input data distribution changes (e.g., customer demographics shift). Detected via statistics on input features.

Model Drift: Model performance on the data degrades, even without distribution shift. Detected via monitoring prediction confidence and actual outcomes.

Concept Drift: The relationship between inputs and outputs changes (e.g., customer behavior patterns evolve). Hardest to detect and handle.

Experiment Tracking with MLflow

MLflow is an open-source platform for managing ML workflows. Its core components are: Tracking, Projects, Models, and Registry.

MLflow Tracking Basics

Log parameters, metrics, and artifacts for every experiment:

Code — MLflow Experiment Tracking
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import pandas as pd

# Load data
X_train = pd.read_csv('train_features.csv')
y_train = pd.read_csv('train_labels.csv').values.ravel()
X_test = pd.read_csv('test_features.csv')
y_test = pd.read_csv('test_labels.csv').values.ravel()

# Start tracking experiment
mlflow.set_experiment("credit-risk-classification")

with mlflow.start_run(run_name="rf-baseline"):
    # Log parameters
    n_estimators = 100
    max_depth = 10
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", max_depth)
    
    # Train model
    model = RandomForestClassifier(
        n_estimators=n_estimators,
        max_depth=max_depth,
        random_state=42
    )
    model.fit(X_train, y_train)
    
    # Evaluate
    y_pred = model.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    
    # Log metrics
    mlflow.log_metric("accuracy", accuracy)
    mlflow.log_metric("f1", f1_score(y_test, y_pred, average='weighted'))
    
    # Log model
    mlflow.sklearn.log_model(model, "model")
    
    # Log artifacts (plots, reports, etc.)
    import matplotlib.pyplot as plt
    from sklearn.metrics import confusion_matrix
    cm = confusion_matrix(y_test, y_pred)
    plt.figure(figsize=(8, 6))
    plt.imshow(cm, cmap='Blues')
    plt.title('Confusion Matrix')
    plt.colorbar()
    plt.savefig('confusion_matrix.png')
    mlflow.log_artifact('confusion_matrix.png')
    
    print(f"Logged experiment with accuracy: {accuracy:.4f}")

Viewing Results

Start the MLflow UI with:

Code — Launch MLflow UI
mlflow ui --host 127.0.0.1 --port 5000

Then visit http://127.0.0.1:5000. You'll see all experiments, runs, parameters, metrics, and artifacts.

MLflow Model Registry

Centralized model management. Register models, manage versions (Staging/Production/Archived), and transition them through lifecycle stages with approval workflows.

Hyperparameter Tuning with MLflow

Combined with Optuna or Ray Tune for automated hyperparameter optimization:

Code — MLflow + Optuna
import optuna
import mlflow

def objective(trial):
    n_estimators = trial.suggest_int("n_estimators", 50, 500)
    max_depth = trial.suggest_int("max_depth", 5, 50)
    min_samples_split = trial.suggest_int("min_samples_split", 2, 20)
    
    with mlflow.start_run():
        mlflow.log_param("n_estimators", n_estimators)
        mlflow.log_param("max_depth", max_depth)
        mlflow.log_param("min_samples_split", min_samples_split)
        
        model = RandomForestClassifier(
            n_estimators=n_estimators,
            max_depth=max_depth,
            min_samples_split=min_samples_split,
            random_state=42
        )
        model.fit(X_train, y_train)
        
        y_pred = model.predict(X_test)
        accuracy = accuracy_score(y_test, y_pred)
        mlflow.log_metric("accuracy", accuracy)
        
        return accuracy

# Run optimization
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=20)

print(f"Best trial: {study.best_trial.number}")
print(f"Best accuracy: {study.best_value:.4f}")

Comparing with Other Tools

MLflow is open-source and self-hosted. Alternatives:

Weights & Biases (W&B): Cloud-based, excellent visualizations, integrates with many frameworks
Neptune: Cloud-based, lightweight, good for large experiments
Comet ML: Cloud-based, free tier available

Containerization with Docker

Docker packages your model and all its dependencies (Python, libraries, system libraries) into a container. Benefits:

Reproducibility: Runs identically on your laptop, staging, and production
Dependency Management: No "works on my machine" problems
Isolation: Multiple containers can run different versions simultaneously
Scalability: Containers are lightweight and start quickly

Building a Model Serving Container

Create a Dockerfile for your model:

Code — Dockerfile for ML Model
FROM python:3.10-slim

WORKDIR /app

# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy model, code, and config
COPY model.pkl .
COPY app.py .

# Expose port for API
EXPOSE 8000

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD python -c "import requests; requests.get('http://localhost:8000/health')"

# Start the API server
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Requirements File

Code — requirements.txt
scikit-learn==1.3.0
numpy==1.24.0
pandas==2.0.0
fastapi==0.104.0
uvicorn==0.24.0
pydantic==2.4.0
python-multipart==0.0.6

Multi-stage Build for Smaller Images

Reduce image size by building in one stage and copying artifacts to a minimal runtime stage:

Code — Multi-stage Dockerfile
# Stage 1: Builder
FROM python:3.10-slim as builder

WORKDIR /build

COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Stage 2: Runtime
FROM python:3.10-slim

WORKDIR /app

# Copy only necessary files from builder
COPY --from=builder /root/.local /root/.local
COPY model.pkl .
COPY app.py .

ENV PATH=/root/.local/bin:$PATH

EXPOSE 8000
HEALTHCHECK --interval=30s CMD python -c "import requests; requests.get('http://localhost:8000/health')"
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Building and Running the Container

Code — Docker Commands
# Build the image
docker build -t credit-risk-model:1.0 .

# Run the container
docker run -p 8000:8000 credit-risk-model:1.0

# Push to registry (Docker Hub, AWS ECR, etc.)
docker tag credit-risk-model:1.0 myrepo/credit-risk-model:1.0
docker push myrepo/credit-risk-model:1.0

# Run with environment variables
docker run -p 8000:8000 \
    -e MODEL_PATH=/models/model.pkl \
    -e LOG_LEVEL=INFO \
    credit-risk-model:1.0

Container Registry Best Practices

Use AWS ECR, Google Artifact Registry, or Docker Hub to store images. Tag images with git commit SHA for full traceability.

Image Optimization

Use slim or alpine base images (50-100MB vs 300MB+)
Combine RUN commands to reduce layers
Use .dockerignore to exclude unnecessary files
Install only runtime dependencies (not dev tools)
Cache layers efficiently (dependencies before code)

Model Serving with FastAPI

FastAPI is a modern Python web framework for building high-performance APIs. Perfect for ML model serving.

Building a Simple Prediction API

Code — FastAPI Model Server
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import pickle
import numpy as np
from typing import List
import uvicorn

# Load model once at startup
with open('model.pkl', 'rb') as f:
    model = pickle.load(f)

app = FastAPI(
    title="Credit Risk Prediction API",
    version="1.0.0",
    docs_url="/docs"  # Swagger UI at /docs
)

# Define request/response schemas
class PredictionRequest(BaseModel):
    age: int = Field(..., ge=18, le=120)
    income: float = Field(..., gt=0)
    credit_score: int = Field(..., ge=300, le=850)
    existing_loans: int = Field(..., ge=0)
    employment_years: float = Field(..., ge=0)

class PredictionResponse(BaseModel):
    prediction: int  # 0 or 1
    probability: float
    risk_level: str

@app.get("/health")
async def health_check():
    """Health check endpoint"""
    return {"status": "healthy"}

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    """Make a prediction"""
    try:
        # Prepare features
        features = np.array([[
            request.age,
            request.income,
            request.credit_score,
            request.existing_loans,
            request.employment_years
        ]])
        
        # Make prediction
        prediction = model.predict(features)[0]
        probability = model.predict_proba(features)[0, prediction]
        
        # Determine risk level
        risk_level = "HIGH" if prediction == 1 else "LOW"
        
        return PredictionResponse(
            prediction=int(prediction),
            probability=float(probability),
            risk_level=risk_level
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.post("/batch-predict")
async def batch_predict(requests: List[PredictionRequest]):
    """Batch predictions"""
    features = np.array([[
        r.age, r.income, r.credit_score, r.existing_loans, r.employment_years
    ] for r in requests])
    
    predictions = model.predict(features)
    probabilities = model.predict_proba(features)[:, 1]
    
    return {
        "predictions": predictions.tolist(),
        "probabilities": probabilities.tolist()
    }

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Advanced Features

Request Validation & Documentation

FastAPI automatically generates OpenAPI documentation (Swagger UI at /docs) from your Pydantic models.

Async Predictions

Use async def for I/O-bound operations (database queries, external API calls):

Code — Async FastAPI Endpoint
@app.post("/async-predict")
async def async_predict(request: PredictionRequest):
    # Async database lookup
    user_history = await db.get_user_history(request.user_id)
    
    # Async external call
    credit_bureaus = await fetch_from_credit_bureau(request.ssn)
    
    # Inference
    features = prepare_features(request, user_history, credit_bureaus)
    prediction = model.predict(features)
    
    return {"prediction": prediction}

Model Caching & Versioning

Code — Multiple Model Versions
import os
from pathlib import Path

# Load multiple models
models = {
    "v1.0": pickle.load(open("models/model_v1.pkl", "rb")),
    "v1.1": pickle.load(open("models/model_v1_1.pkl", "rb")),
    "v2.0": pickle.load(open("models/model_v2.pkl", "rb")),
}

@app.post("/predict")
async def predict(request: PredictionRequest, model_version: str = "v2.0"):
    if model_version not in models:
        raise HTTPException(status_code=404, detail="Model version not found")
    
    model = models[model_version]
    features = prepare_features(request)
    prediction = model.predict(features)
    
    return {"prediction": prediction, "model_version": model_version}

Production Serving Platforms

TorchServe: Purpose-built for PyTorch models, with batching and A/B testing
KServe: Kubernetes-native model serving (InferenceService CRD)
Seldon Core: Kubernetes serving with advanced deployment patterns
Ray Serve: Distributed Python serving with auto-scaling
Cloud Functions: AWS Lambda, Google Cloud Functions (serverless)

CI/CD Pipelines with GitHub Actions

CI/CD (Continuous Integration / Continuous Deployment) automates testing, building, and deploying your models. GitHub Actions is a free, built-in solution for GitHub repos.

ML Pipeline Workflow

A typical ML CI/CD pipeline includes:

Trigger: Push to main branch
Install: Set up environment and dependencies
Test: Run unit tests on model code
Train: Train model on test dataset
Validate: Check metrics meet thresholds
Build: Create Docker image
Push: Push image to registry
Deploy: Deploy to staging/production

GitHub Actions Workflow File

Code — .github/workflows/ml-pipeline.yaml
name: ML Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          cache: 'pip'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest pytest-cov
      
      - name: Lint with flake8
        run: |
          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
      
      - name: Run unit tests
        run: pytest tests/ --cov --cov-report=xml
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage.xml

  train:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          cache: 'pip'
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Train model
        run: python train.py
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }}
      
      - name: Validate metrics
        run: |
          python validate.py
          if [ $? -ne 0 ]; then
            echo "Model metrics below threshold"
            exit 1
          fi
      
      - name: Upload model artifact
        uses: actions/upload-artifact@v3
        with:
          name: model
          path: models/model.pkl

  build:
    runs-on: ubuntu-latest
    needs: train
    if: github.ref == 'refs/heads/main'
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Log in to Container Registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Download model artifact
        uses: actions/download-artifact@v3
        with:
          name: model
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    runs-on: ubuntu-latest
    needs: build
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to Kubernetes
        run: |
          mkdir -p $HOME/.kube
          echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > $HOME/.kube/config
          
          # Update image in deployment
          kubectl set image deployment/model-server \
            model=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
            -n production
          
          # Wait for rollout
          kubectl rollout status deployment/model-server -n production

Training Script (train.py)

Code — train.py
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score
import pandas as pd
import os

# Load data
df = pd.read_csv('data/train.csv')
X = df.drop('target', axis=1)
y = df['target']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# MLflow setup
mlflow.set_experiment("credit-risk-production")

with mlflow.start_run():
    # Hyperparameters
    params = {
        "n_estimators": 100,
        "max_depth": 15,
        "min_samples_split": 5,
        "random_state": 42
    }
    
    for key, value in params.items():
        mlflow.log_param(key, value)
    
    # Train
    model = RandomForestClassifier(**params)
    model.fit(X_train, y_train)
    
    # Evaluate
    y_pred = model.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred, average='weighted')
    
    mlflow.log_metric("accuracy", accuracy)
    mlflow.log_metric("f1", f1)
    
    # Save model
    mlflow.sklearn.log_model(model, "model")
    
    print(f"Training complete. Accuracy: {accuracy:.4f}")

Validation Script (validate.py)

Code — validate.py
import mlflow
import mlflow.sklearn
import json

# Load latest model run
client = mlflow.tracking.MlflowClient()
experiment = client.get_experiment_by_name("credit-risk-production")
runs = client.search_runs(experiment.experiment_id, order_by=["start_time DESC"])
latest_run = runs[0]

# Get metrics
metrics = latest_run.data.metrics

# Check thresholds
MIN_ACCURACY = 0.85
MIN_F1 = 0.80

if metrics['accuracy'] < MIN_ACCURACY:
    print(f"FAIL: Accuracy {metrics['accuracy']:.4f} < {MIN_ACCURACY}")
    exit(1)

if metrics['f1'] < MIN_F1:
    print(f"FAIL: F1 {metrics['f1']:.4f} < {MIN_F1}")
    exit(1)

print("PASS: All metrics pass thresholds")
print(json.dumps(metrics, indent=2))

Kubernetes Deployment at Scale

Kubernetes orchestrates containerized applications, handling deployment, scaling, and networking. Essential for production ML systems.

Deployment YAML

Code — kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: credit-risk-model
  namespace: production
  labels:
    app: credit-risk-model
    version: v1.0
spec:
  replicas: 3
  selector:
    matchLabels:
      app: credit-risk-model
  template:
    metadata:
      labels:
        app: credit-risk-model
        version: v1.0
    spec:
      containers:
      - name: model
        image: ghcr.io/company/credit-risk-model:1.0.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 8000
        
        # Resource requests and limits
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        
        # Environment variables
        env:
        - name: LOG_LEVEL
          value: "INFO"
        - name: MODEL_VERSION
          value: "1.0.0"
        - name: MAX_BATCH_SIZE
          value: "100"
        
        # Health checks
        livenessProbe:
          httpGet:
            path: /health
            port: http
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        
        readinessProbe:
          httpGet:
            path: /health
            port: http
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2
        
        # Graceful shutdown
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
      
      # Pod disruption budget
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - credit-risk-model
              topologyKey: kubernetes.io/hostname

Service YAML

Code — kubernetes/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: credit-risk-model
  namespace: production
spec:
  selector:
    app: credit-risk-model
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: http
    protocol: TCP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

Horizontal Pod Autoscaler

Code — kubernetes/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: credit-risk-model-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: credit-risk-model
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
      selectPolicy: Max

Applying to Kubernetes

Code — kubectl Commands
# Apply configuration
kubectl apply -f kubernetes/

# Check status
kubectl get deployment credit-risk-model -n production
kubectl get pods -n production -l app=credit-risk-model

# View logs
kubectl logs -f deployment/credit-risk-model -n production

# Port forward for local testing
kubectl port-forward service/credit-risk-model 8000:80 -n production

# Rolling update (change image tag in deployment.yaml first)
kubectl apply -f kubernetes/deployment.yaml

# Rollback if needed
kubectl rollout history deployment/credit-risk-model -n production
kubectl rollout undo deployment/credit-risk-model -n production

KServe for Advanced Serving

KServe provides a Kubernetes-native way to deploy, monitor, and manage ML models with advanced features like traffic splitting and canary deployments:

Code — kubernetes/kserve-inference.yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: credit-risk-model
  namespace: production
spec:
  predictor:
    minReplicas: 3
    maxReplicas: 10
    sklearnserver:
      storageUri: s3://model-registry/credit-risk-model.pkl
      resources:
        requests:
          memory: "512Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "1000m"
  transformer:
    image: company/feature-transformer:v1
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
  canaryTrafficPercent: 10

Monitoring and Observability

Production models fail silently. A model that returns confident predictions on out-of-distribution data is worse than no model. Comprehensive monitoring is essential.

Key Metrics to Monitor

Latency

P50, P95, P99 response times. Target <100ms for real-time models. Monitor database query time, model inference time, and network overhead separately.

Throughput

Requests per second. Track separately for batch and real-time. Monitor queue depth and processing time.

Error Rate

Percentage of failed predictions. Alert if >0.1%. Track by error type (timeout, OOM, invalid input, etc.).

Model Performance

Track prediction confidence, positive class ratio. Alert on sudden changes.

Prometheus Metrics

Export metrics from your model server using Prometheus client:

Code — Prometheus Metrics in FastAPI

from prometheus_client import Counter, Histogram, Gauge, generate_latest
from fastapi import Response
import time

# Define metrics
request_count = Counter(
    'model_predictions_total',
    'Total predictions',
    ['model_version', 'status']
)

prediction_latency = Histogram(
    'model_prediction_duration_seconds',
    'Prediction latency',
    ['model_version'],
    buckets=(0.01, 0.05, 0.1, 0.5, 1.0)
)

prediction_confidence = Histogram(
    'model_prediction_confidence',
    'Prediction confidence distribution',
    buckets=(0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.99)
)

model_errors = Counter(
    'model_errors_total',
    'Total model errors',
    ['error_type']
)

active_requests = Gauge(
    'model_active_requests',
    'Currently processing requests'
)

@app.post("/predict")
async def predict(request: PredictionRequest):
    active_requests.inc()
    start = time.time()
    
    try:
        features = prepare_features(request)
        prediction = model.predict(features)[0]
        probability = model.predict_proba(features).max()
        
        latency = time.time() - start
        prediction_latency.labels(model_version="1.0").observe(latency)
        prediction_confidence.observe(probability)
        request_count.labels(model_version="1.0", status="success").inc()
        
        return {"prediction": prediction, "confidence": probability}
    except Exception as e:
        model_errors.labels(error_type=type(e).__name__).inc()
        request_count.labels(model_version="1.0", status="error").inc()
        raise
    finally:
        active_requests.dec()

@app.get("/metrics")
async def metrics():
    return Response(generate_latest(), media_type="text/plain")

Dashboard Example (Grafana)

Configure Grafana to query Prometheus. Key dashboard panels:

Request Rate (QPS): rate(model_predictions_total[1m])
Error Rate: rate(model_errors_total[1m])
P95 Latency: histogram_quantile(0.95, model_prediction_duration_seconds)
Median Confidence: histogram_quantile(0.5, model_prediction_confidence)

Alerting Rules

Code — prometheus-alerts.yml
groups:
- name: model-alerts
  interval: 30s
  rules:
  - alert: HighErrorRate
    expr: rate(model_errors_total[5m]) > 0.01
    for: 5m
    annotations:
      summary: "High error rate detected"
      description: "Error rate is {{ $value | humanizePercentage }}"
  
  - alert: HighLatency
    expr: histogram_quantile(0.95, rate(model_prediction_duration_seconds_bucket[5m])) > 0.5
    for: 5m
    annotations:
      summary: "Model latency too high"
      description: "P95 latency is {{ $value }}s"
  
  - alert: ZeroThroughput
    expr: rate(model_predictions_total[1m]) == 0
    for: 2m
    annotations:
      summary: "Model receiving no traffic"
  
  - alert: LowPredictionConfidence
    expr: histogram_quantile(0.5, model_prediction_confidence) < 0.6
    for: 10m
    annotations:
      summary: "Model confidence dropped"

Data Drift Detection with Evidently

Data drift occurs when the distribution of input features changes over time. Evidently detects drift, monitors model performance, and generates reports.

Evidently Setup

Code — Data Drift Detection
import pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, RegressionPreset
from evidently.metrics import (
    DataDriftTable,
    ColumnDriftMetric,
    DatasetMissingValuesMetric,
)
import json

# Load reference (training) and current (production) data
reference_data = pd.read_csv('data/train.csv')
production_data = pd.read_csv('data/production_batch.csv')

# Create drift report
drift_report = Report(metrics=[
    DataDriftPreset(),
    DatasetMissingValuesMetric(),
])

drift_report.run(reference_data=reference_data, current_data=production_data)

# Get results
results = drift_report.as_dict()
drifted_columns = [
    col for col in results['metrics'][0]['result']['drift_by_columns']
    if results['metrics'][0]['result']['drift_by_columns'][col]['drift_detected']
]

print(f"Drifted columns: {drifted_columns}")

# Save report
drift_report.save_html('drift_report.html')

# Alert if significant drift detected
if len(drifted_columns) > 3:
    send_alert(f"High drift detected: {drifted_columns}")

Model Performance Monitoring

Code — Model Performance Report
from evidently.report import Report
from evidently.metric_preset import RegressionPreset, ClassificationPreset

# For classification
perf_report = Report(metrics=[ClassificationPreset()])

perf_report.run(
    reference_data=reference_data,
    current_data=production_data,
    column_mapping={
        "target": "y_true",
        "prediction": "y_pred"
    }
)

perf_report.save_html('performance_report.html')

# Extract metrics
metrics = perf_report.as_dict()
current_accuracy = metrics['metrics'][0]['result']['metrics']['accuracy']
current_f1 = metrics['metrics'][0]['result']['metrics']['f1']

print(f"Current accuracy: {current_accuracy:.4f}")
print(f"Current F1: {current_f1:.4f}")

Continuous Monitoring Pipeline

Code — Daily Drift Check
import schedule
import time
from datetime import datetime, timedelta
import logging

logger = logging.getLogger(__name__)

def check_drift():
    """Run daily drift check"""
    try:
        # Load reference data (monthly retraining baseline)
        reference_date = datetime.now() - timedelta(days=30)
        reference_data = load_from_database(
            start_date=reference_date - timedelta(days=30),
            end_date=reference_date
        )
        
        # Load current data (last 24 hours)
        current_data = load_from_database(
            start_date=datetime.now() - timedelta(hours=24),
            end_date=datetime.now()
        )
        
        # Run drift detection
        drift_report = Report(metrics=[DataDriftPreset()])
        drift_report.run(reference_data=reference_data, current_data=current_data)
        
        # Check results
        results = drift_report.as_dict()
        drifted_columns = [
            col for col in results['metrics'][0]['result']['drift_by_columns']
            if results['metrics'][0]['result']['drift_by_columns'][col]['drift_detected']
        ]
        
        # Alert if necessary
        if len(drifted_columns) > 3:
            logger.warning(f"Drift detected in {len(drifted_columns)} columns: {drifted_columns}")
            send_slack_alert(
                f"Data drift alert: {drifted_columns}\n"
                f"Recommend model retraining"
            )
        else:
            logger.info("No significant drift detected")
        
        # Save report to monitoring database
        save_report_to_db(drift_report, datetime.now())
        
    except Exception as e:
        logger.error(f"Drift check failed: {e}")
        send_slack_alert(f"Drift check failed: {e}")

# Schedule daily check
schedule.every().day.at("02:00").do(check_drift)

while True:
    schedule.run_pending()
    time.sleep(60)

Great Expectations Integration

Great Expectations provides data quality checks. Use it to validate that production data meets expectations:

Code — Great Expectations Validation
from great_expectations.dataset import PandasDataset
import great_expectations as ge

# Create expectation suite
context = ge.get_context()
batch = context.get_batch(
    datasource_name="production_db",
    data_connector_name="daily_data",
    data_asset_name="predictions"
)

# Add expectations
batch.expect_column_values_to_be_between('age', min_value=18, max_value=120)
batch.expect_column_values_to_be_in_set('gender', value_set=['M', 'F'])
batch.expect_column_values_to_not_be_null('income')
batch.expect_column_kl_divergence_from_list(
    'credit_score',
    partition_object={'column': 'date', 'value': '2024-01-01'}
)

# Run validation
checkpoint = context.add_checkpoint(
    name="production_validation",
    validations=[
        {
            "batch_request": batch_request,
            "expectation_suite_name": "production_suite"
        }
    ]
)

result = checkpoint.run()

if not result.success:
    logger.warning(f"Validation failed: {result}")
    send_alert("Data quality check failed")

MLOps Tool Comparison

Experiment Tracking Tools

Tool	Hosting	Cost	Best For	Community
MLflow	Self-hosted or cloud	Free (open-source)	Enterprise, full control	Large, growing
Weights & Biases	Cloud only	Freemium	Research, beautiful UI	Large, academic
Neptune	Cloud	Freemium	Production teams, lightweight	Growing
Comet ML	Cloud	Freemium	Team collaboration	Small-medium

Model Registry Comparison

Tool	Versioning	Staging Support	Approval Workflows	Integration
MLflow Model Registry	Semantic versioning	Staging/Production/Archived	Manual approval	Tight with MLflow
HuggingFace Hub	Git-based	Not built-in	Not built-in	HF ecosystem
AWS SageMaker Model Registry	Model package versioning	Development/Approved/Production	AWS approval	AWS services
Google Vertex AI Registry	Full version history	Not clearly separated	Not built-in	GCP services

Model Serving Platforms

Tool	Framework Support	Scaling	Latency	Ease of Use
FastAPI + Uvicorn	Any (custom)	Manual Kubernetes	Low (20-50ms)	High
TorchServe	PyTorch	Built-in batching & scaling	Low	Medium
KServe	Any (via containers)	Auto-scaling, GPU	Low	Medium
AWS SageMaker	Any	Auto-scaling, serverless	Medium	Easy (AWS native)
Ray Serve	Any	Automatic	Low	Medium

MLOps Maturity Model

Organizations mature through five stages of MLOps capability. Understanding your current stage helps prioritize improvements.

Level 1: Ad Hoc

Manual experiments, no tracking, notebooks, no version control

Level 1

Characteristics: Models in Jupyter notebooks, manual deployment, no monitoring. Data scientists work in silos. "It works on my machine."

Pain points: Impossible to reproduce results, high error rates in production, no accountability.

Level 2: Organized

Git repos, basic experiment tracking, documented processes

Level 2

Characteristics: Code is versioned, experiments are tracked with MLflow, but deployments are manual. Test/production environments exist but are inconsistent.

Pain points: Slow deployments (days), training-serving skew, limited monitoring.

Level 3: Operationalized

Automated CI/CD, containerization, basic monitoring, model registry

Level 3

Characteristics: Models are containerized and deployed via CI/CD. Model registry tracks versions. Automated tests ensure code quality. Basic monitoring alerts on obvious failures.

Pain points: Retraining is still manual, limited drift detection, no feature store.

Level 4: Advanced

Automated retraining, data drift detection, feature store, A/B testing

Level 4

Characteristics: Retraining pipelines are automated on schedules or drift triggers. A feature store provides consistent features. Data drift is detected and alerts team. A/B testing infrastructure supports canary deployments.

Pain points: Complex orchestration, storage costs, talent bottleneck.

Level 5: World-Class

End-to-end automation, multi-model systems, zero-downtime updates

Level 5

Characteristics: Entire ML system is fully automated from data ingestion through deployment. Multiple models work together (ensemble, cascading). Updates occur with zero downtime. Self-healing systems recover from failures automatically.

Examples: Google, Meta, OpenAI production systems.

MLOps Best Practices

Version Everything

Code, data, models, and configuration. Use git for code/config, data versioning tools for datasets (DVC, Delta Lake), MLflow for models.

Automate Testing

Unit tests for data processing, integration tests for pipelines, property tests for model outputs. Aim for >80% code coverage.

Containerize Everything

Docker ensures reproducibility across environments. Use multi-stage builds to minimize image size.

Monitor Continuously

Track latency, throughput, error rates, and model performance metrics. Alert on anomalies.

Document Everything

README for datasets, docstrings for functions, runbooks for deployments. Future you will thank present you.

Use Feature Stores

Centralize feature engineering. Prevents training-serving skew and reduces redundant computation.

Common Anti-Patterns to Avoid

Training on All Data

Always hold out a test set. Better: use time-based splits (train on past, test on future). Never touch test data during development.

Ignoring Data Drift

Models degrade silently when data distribution shifts. Monitor drift continuously and retrain when detected.

No Monitoring

"Silent failures" are common. A model confidently predicting on out-of-distribution data can cause massive damage.

Manual Deployments

Error-prone and slow. Automate everything with CI/CD. Make deployments boring.

Treating Infrastructure as Secondary

The infrastructure is as important as the model. Poor infrastructure causes production outages, data quality issues, and slow deployments.

MLOps Architecture Patterns

Batch Prediction Pipeline

Process large datasets periodically (hourly, daily).

Raw Data → Feature Engineering → Model Inference → Results Storage → Reporting
    ↓               ↓                   ↓              ↓
  S3/DW      Feature Store       Model Registry    Database

Use cases: Batch credit scoring, daily churn predictions, nightly recommendations.

Real-time Serving Architecture

REST Request → Load Balancer → Model Replicas → Result
                                     ↓
                            Shared Model Cache
                                     ↓
                            Monitoring/Logging

Use cases: Fraud detection API, recommendation engine, real-time bidding.

Feature Store Architecture

Raw Data Sources
    ↓
Feature Computation Layer
    ↓
Feature Store (Offline + Online)
    ↓
Model Training (Offline) AND Model Serving (Online)

Feature Store enables consistent features between training and serving, preventing skew.

Multi-Model Ensemble

Request
  ↓
Router (decides which model(s) to use)
  ↓
Model A  Model B  Model C
  ↓        ↓        ↓
Ensemble/Voting
  ↓
Response

Combine multiple models for robustness and improved performance.

Complete Code Examples

End-to-End MLOps Pipeline

Code — Complete ML Pipeline Script
#!/usr/bin/env python3
"""End-to-end MLOps pipeline: data → model → deployment"""

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
import mlflow
import mlflow.sklearn
import pickle
import logging
from datetime import datetime
import yaml
import argparse

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def load_config(config_path):
    """Load configuration from YAML"""
    with open(config_path) as f:
        return yaml.safe_load(f)

def load_data(config):
    """Load and preprocess data"""
    logger.info("Loading data...")
    df = pd.read_csv(config['data']['train_path'])
    
    # Handle missing values
    df = df.dropna()
    
    # Remove outliers
    numeric_cols = df.select_dtypes(include=[np.number]).columns
    for col in numeric_cols:
        Q1 = df[col].quantile(0.25)
        Q3 = df[col].quantile(0.75)
        IQR = Q3 - Q1
        df = df[(df[col] >= Q1 - 1.5*IQR) & (df[col] <= Q3 + 1.5*IQR)]
    
    return df

def prepare_features(df, target_col):
    """Prepare features and target"""
    X = df.drop(target_col, axis=1)
    y = df[target_col]
    return X, y

def train_model(X_train, y_train, config):
    """Train model"""
    logger.info("Training model...")
    params = config['model']['params']
    model = RandomForestClassifier(**params, random_state=42)
    model.fit(X_train, y_train)
    return model

def evaluate_model(model, X_test, y_test):
    """Evaluate model performance"""
    y_pred = model.predict(X_test)
    y_pred_proba = model.predict_proba(X_test)[:, 1]
    
    metrics = {
        'accuracy': accuracy_score(y_test, y_pred),
        'f1': f1_score(y_test, y_pred, average='weighted'),
        'roc_auc': roc_auc_score(y_test, y_pred_proba),
    }
    
    return metrics, y_pred

def main(config_path):
    config = load_config(config_path)
    
    # MLflow setup
    mlflow.set_experiment(config['mlflow']['experiment_name'])
    
    with mlflow.start_run(run_name=f"run-{datetime.now().strftime('%Y%m%d-%H%M%S')}"):
        # Load and prepare data
        df = load_data(config)
        X, y = prepare_features(df, config['data']['target_col'])
        
        X_train, X_test, y_train, y_test = train_test_split(
            X, y, test_size=config['data']['test_size'], random_state=42
        )
        
        # Scale features
        scaler = StandardScaler()
        X_train_scaled = scaler.fit_transform(X_train)
        X_test_scaled = scaler.transform(X_test)
        
        # Train model
        model = train_model(X_train_scaled, y_train, config)
        
        # Evaluate
        metrics, y_pred = evaluate_model(model, X_test_scaled, y_test)
        
        # Log to MLflow
        mlflow.log_params(config['model']['params'])
        for metric_name, metric_value in metrics.items():
            mlflow.log_metric(metric_name, metric_value)
        
        # Save artifacts
        mlflow.sklearn.log_model(model, "model")
        
        # Save scaler
        with open('scaler.pkl', 'wb') as f:
            pickle.dump(scaler, f)
        mlflow.log_artifact('scaler.pkl')
        
        logger.info(f"Metrics: {metrics}")
        
        # Check thresholds
        if metrics['accuracy'] >= config['model']['min_accuracy']:
            logger.info("Model passed validation!")
        else:
            logger.error(f"Model failed: accuracy {metrics['accuracy']} < {config['model']['min_accuracy']}")
            exit(1)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--config", default="config.yaml")
    args = parser.parse_args()
    main(args.config)

Configuration File (config.yaml)

Code — config.yaml
data:
  train_path: "data/train.csv"
  test_size: 0.2
  target_col: "target"

model:
  params:
    n_estimators: 100
    max_depth: 15
    min_samples_split: 5
  min_accuracy: 0.85

mlflow:
  experiment_name: "credit-risk-production"
  tracking_uri: "http://localhost:5000"

Hands-on Exercises

Exercise 1: Set Up MLflow Experiment Tracking

Goal: Track 5 model experiments with different hyperparameters.

Steps:

Install MLflow: pip install mlflow
Create a training script that trains 5 RandomForest models with different max_depth values (5, 10, 15, 20, 25)
Log parameters and metrics for each run
Launch MLflow UI and compare experiments
Identify which hyperparameters gave best results

Exercise 2: Containerize a Model Server

Goal: Create a Docker image for a FastAPI model server.

Steps:

Create a FastAPI endpoint that loads a pre-trained model and serves predictions
Write a Dockerfile
Build the image: docker build -t mymodel:1.0 .
Run the container: docker run -p 8000:8000 mymodel:1.0
Test the API with curl: curl -X POST http://localhost:8000/predict -d '{...}'

Exercise 3: Deploy to Kubernetes

Goal: Deploy the containerized model to Kubernetes with auto-scaling.

Steps:

Push Docker image to a registry (Docker Hub, ECR, etc.)
Create Deployment and Service YAML files
Deploy: kubectl apply -f deployment.yaml service.yaml
Create HPA for auto-scaling
Generate load and watch it scale: kubectl get hpa -w

Exercise 4: Detect Data Drift

Goal: Use Evidently to detect drift in production data.

Steps:

Install Evidently: pip install evidently
Load reference data (training data) and production data
Create a DataDriftPreset report
Identify which columns show drift
Generate an HTML report and visualize the drift

Interview Questions

These questions test your understanding of production ML systems and are commonly asked in interviews.

Q1: Explain the difference between data drift and model drift. ▼

Data Drift: The distribution of input features changes. Example: customer income distribution shifts after a recession. Detected via statistical tests on input features.

Model Drift: Model performance degrades on current data. Can happen with or without data drift. Detected via monitoring prediction metrics and comparing to holdout performance.

Concept Drift: The relationship between inputs and outputs changes. Example: customer preferences shift. Hardest to detect — requires outcome monitoring.

Q2: You deployed a model to production and accuracy dropped from 92% to 78% overnight. What could cause this? ▼

Possible causes: (1) Data drift — input distribution changed. (2) Missing data — pipeline started returning nulls. (3) Data quality issue — garbage in, garbage out. (4) Infrastructure problem — wrong model was deployed. (5) Concept drift — the problem itself changed.

Debugging approach: Check prediction distribution, analyze input feature distributions, compare current data to training data, look at logs for errors, monitor latency/throughput. Have alerts on these metrics to catch issues faster.

Q3: How do you handle the training-serving skew problem? ▼

Training-serving skew occurs when features are computed differently during training vs serving. Solutions: (1) Use a feature store (Feast, Tecton) to compute features once and share. (2) Version features and track lineage. (3) Unit test that training and serving code compute identical features. (4) Use the same feature library in both pipelines. (5) Log features used in production and compare to training distribution.

Q4: Design a system to retrain models automatically when drift is detected. ▼

Architecture:

(1) Drift Detection: Daily batch job compares current data to baseline using Evidently or Great Expectations. If drift detected, publish event.

(2) Retraining Trigger: Listen for drift events. If significant drift (e.g., >3 features), trigger retraining pipeline.

(3) Retraining: Async job trains new model with recent data. Logs metrics to MLflow.

(4) Validation: Automatically validate that new model outperforms old model on holdout test set.

(5) Deployment: If validation passes, deploy new model. If fails, alert humans.

Challenges: Concept drift can't be detected without outcome labels. Need to monitor prediction feedback loop.

Q5: How do you A/B test models in production? ▼

Setup: Route traffic between control (old model) and treatment (new model). Typically 90% old, 10% new initially.

Metrics: Track business metrics (revenue, conversion, retention), not just ML metrics (accuracy). Use statistical tests (t-test, chi-squared) to determine if difference is significant.

Duration: Run for at least 7 days to capture weekly patterns, or until statistical significance is reached.

Rollout: If treatment wins, gradually increase traffic to new model (50%, then 100%). If fails, rollback immediately.

Tools: Kubernetes service mesh (Istio) for traffic splitting, Argo for progressive deployments.

Q6: What metrics should you monitor for a production ML system? ▼

System Metrics:

Latency (P50, P95, P99)
Throughput (requests/second)
Error rate
Resource usage (CPU, memory, GPU)

Model Metrics:

Prediction confidence distribution
Positive class ratio (for classification)
Feature distributions
Model performance (if outcomes available)

Business Metrics:

Revenue impact
Conversion rate
User satisfaction

Q7: How do you version models and manage deployments? ▼

Versioning Strategy: Store model artifacts in a registry (MLflow, Hugging Face, cloud provider). Tag each version with: code commit SHA, training data hash, hyperparameters, metrics, timestamp.

Deployment Stages: Development (on-demand) → Staging (auto-scaled) → Canary (10% traffic) → Production (100% traffic).

Rollback: Keep previous model version available for quick rollback if new version fails.

Tools: MLflow Model Registry, ArgoCD, Spinnaker for deployment orchestration.

Q8: Explain the MLOps maturity model and where most companies are. ▼

Level 1 (Ad Hoc): Notebooks, manual deployments. Very common in startups.

Level 2 (Organized): Git repos, experiment tracking, manual deployments. Common in growing companies.

Level 3 (Operationalized): CI/CD, containerization, monitoring. Target for most organizations.

Level 4 (Advanced): Automated retraining, feature store, A/B testing. Companies with mature ML practices.

Level 5 (World-Class): End-to-end automation, self-healing. Only Google, Meta, etc.

Reality: Most companies are stuck at Level 2. The jump to Level 3 requires significant infrastructure investment but pays off quickly.

Frequently Asked Questions

What's the minimum infrastructure needed to do MLOps? ▼

Bare minimum: Git (versioning), a monitoring tool (Prometheus/Grafana), and one server to deploy to.

Realistic minimum: Git, Docker, Kubernetes cluster (managed: EKS/GKE), CI/CD runner (GitHub Actions, GitLab CI), and experiment tracker (MLflow). Budget: $500-2000/month for infrastructure.

How often should we retrain models? ▼

It depends on drift: (1) Stable domains: retrain monthly or quarterly. (2) Drifting domains: retrain weekly or on-demand when drift detected. (3) Fast-moving domains: retrain daily.

Best approach: monitor drift continuously and retrain when detected, rather than on a fixed schedule.

Should we use cloud or on-premises infrastructure? ▼

Cloud (AWS, GCP, Azure): Easier to scale, less ops burden, higher cost at scale. Good for startups and companies without data center expertise.

On-premises: Lower long-term cost at scale, full control, requires infrastructure expertise. Good for enterprises with existing data centers.

Hybrid: Keep sensitive data on-premises, use cloud for compute-heavy training.

How do we prevent data leakage in ML pipelines? ▼

Data leakage occurs when information from test/future data leaks into training. Prevention:

(1) Temporal split: train on past data, test on future. Never shuffle time-series.

(2) Group split: For grouped data (users, sessions), ensure groups don't split across train/test.

(3) Feature engineering: Compute aggregations using only training data, then apply to test data. Use the same normalizers/scalers fitted on training data.

(4) Code review: Have another engineer review pipeline code specifically for leakage.

How do we handle model explainability in production? ▼

Tools: SHAP, LIME, or model-specific explanations (feature importance for trees, attention weights for transformers).

Serve explanations alongside predictions: Many regulations (GDPR, Fair Lending) require explaining predictions. Log explanations to enable audits.

Challenge: Explanations are expensive to compute. Cache or pre-compute where possible.

What's the ROI on MLOps infrastructure investment? ▼

Hard benefits: Faster deployments (hours vs weeks), fewer production failures, reduced debugging time.

Soft benefits: Team alignment, reproducibility, confidence in models.

Typical payback: 6-12 months for teams with 3+ data scientists. The investment in infrastructure pays for itself through efficiency gains.

How do we ensure model fairness in production? ▼

Monitoring: Track model performance separately for demographic groups. Alert if accuracy drops significantly for any group.

Fairness constraints: Train models with fairness objectives (demographic parity, equal opportunity) rather than just accuracy.

Regular audits: Quarterly fairness reviews of model behavior across populations.

Tools: Fairness Toolkit (Microsoft), Aequitas, Responsible AI (Google).

What's the best tool stack for a startup? ▼

Development: Jupyter, DVC (data versioning), Git.

Experiment tracking: MLflow (self-hosted) or Weights & Biases (cloud).

Serving: FastAPI + Docker + Kubernetes (or cloud-managed options).

Monitoring: Prometheus + Grafana, or cloud-native solutions.

Feature store: Start without one (too complex early on), add later if needed.

Cost-saving: Use GitHub Actions (free), managed Kubernetes (cheaper than self-managed), and open-source tools where possible.

How do we handle model serving during traffic spikes? ▼

Auto-scaling: Use Horizontal Pod Autoscaler (Kubernetes) to add replicas when load increases. Typically scales up in seconds.

Load balancing: Distribute requests across replicas with round-robin or least-connections algorithm.

Caching: Cache frequent predictions to reduce model invocations. Be careful about stale predictions.

Queueing: If you can accept some latency, queue requests and process in batches (better GPU utilization).

What are the biggest MLOps challenges we'll face? ▼

1. Data quality: Garbage in, garbage out. Data quality directly impacts model quality. No shortcut.

2. Reproducibility: Ensuring results are reproducible across teams and time requires discipline with versioning.

3. Monitoring: Knowing when models fail. Silent failures are the worst.

4. Organizational: Getting teams to adopt practices requires culture change, not just tools.

5. Cost: ML infrastructure is expensive. Storage, compute, and monitoring costs add up quickly.

Summary and Key Takeaways

MLOps is Essential

Building a model is 5% of the work. The remaining 95% is infrastructure, monitoring, and maintenance. MLOps bridges this gap.

Version Everything

Code, data, models, and config must be versioned. Use git for code, MLflow for models, DVC for data.

Automate the Pipeline

Trigger training and deployment automatically. Use CI/CD to automate testing, building, and deployment.

Monitor Continuously

Track system metrics (latency, throughput), model metrics (confidence, drift), and business metrics. Alert on anomalies.

Expect Drift

Data distribution will change. Monitor drift continuously and retrain when detected.

Tools Are Secondary

The specific tools matter less than having solid engineering practices. Start simple, add tools as you grow.

The MLOps Journey

Most teams follow a similar path:

Phase 1 (Chaos): Notebooks, manual experiments, frustrated data scientists. 3-6 months.
Phase 2 (Organization): Git repos, experiment tracking, documented processes. Deployments still manual. 6-12 months.
Phase 3 (Automation): CI/CD pipelines, containerization, monitoring. Models deploy to production reliably. 12-24 months.
Phase 4 (Intelligence): Automated retraining, drift detection, feature stores. Models heal themselves. 24+ months.

The jump from Phase 2 to Phase 3 is transformative. Suddenly, data scientists can focus on model quality instead of ops. The investment in infrastructure pays for itself immediately.

Resources and Further Learning

Essential Reading

ML Engineering for Production

Andrew Ng's course on Coursera. Covers data pipelines, experiment tracking, deployment. Highly recommended.

Building Machine Learning Systems

Willi Richert & Luis Pedro Coelho. Best practical guide for production ML systems.

Machine Learning Operations (MLOps)

Google Cloud's MLOps guide. Free, comprehensive, cloud-agnostic.

MLOps.community

Community resources, case studies, and best practices. Active Slack community.

Tools and Platforms

Experiment Tracking: MLflow, Weights & Biases, Neptune
Feature Stores: Feast, Tecton, Databricks Feature Store
Data Versioning: DVC, Delta Lake, Iceberg
Model Serving: FastAPI, TorchServe, KServe, Seldon Core
Monitoring: Prometheus, Grafana, DataDog, New Relic
Data Drift: Evidently, Great Expectations, Whylabs
Model Registry: MLflow, HuggingFace Hub, cloud providers

Community and Events

MLOps.community: Online community for practitioners. Slack, forums, weekly discussions.
MLOps World: Annual conference with talks on production ML systems.
Women in ML Ops: Community focusing on diversity in MLOps.
GitHub: Explore MLOps projects and examples on GitHub.

Next Steps

To get started with MLOps:

Set up experiment tracking with MLflow on your next project
Containerize your model with Docker
Create a simple CI/CD pipeline with GitHub Actions
Deploy to Kubernetes or a cloud provider
Add monitoring with Prometheus and Grafana
Implement drift detection with Evidently
Iterate and improve based on your specific needs