Building a Local RAG Pipeline
From Scratch

What is RAG?

Retrieval-Augmented Generation (RAG) is an AI architecture pattern that gives a large language model (LLM) access to a private knowledge base at query time — without retraining or fine-tuning the model itself.

Instead of relying solely on knowledge baked into the model during training, RAG retrieves relevant text chunks from your own documents and injects them directly into the LLM's prompt as context. The model then synthesises an answer grounded in your data.

The Core Idea

Think of RAG like an open-book exam. The LLM is the student — smart, articulate, capable of reasoning — but it is allowed to look at your documents before answering. Without RAG, the student relies on memory alone (training data). With RAG, the student looks up the relevant pages first, then writes the answer.

  User Question
       │
       ▼
  ┌──────────────┐     embed      ┌─────────────────┐
  │  Your Query  │ ────────────► │  Vector Store   │
  └──────────────┘               │  (Qdrant)       │
                                  └────────┬────────┘
                                           │ top-k similar chunks
                                           ▼
  ┌──────────────────────────────────────────────────┐
  │  PROMPT =  System instruction                    │
  │         +  Retrieved chunks  (context)           │
  │         +  User question                         │
  └──────────────────────┬───────────────────────────┘
                         │
                         ▼
                  ┌─────────────┐
                  │  LLM        │  (llama3.1:8b via Ollama)
                  │  Answer     │
                  └─────────────┘

RAG vs Fine-Tuning vs Base LLM

Why Run Locally?

Most RAG tutorials use the OpenAI API — fast to prototype, but every document chunk gets sent to a remote server. For accountants, law firms, medical practices, and any organisation handling sensitive data, that is not acceptable.

Our implementation runs entirely on a Mac mini M4 using Apple Metal GPU acceleration. Nothing leaves the machine.

System Architecture

The system has two distinct pipelines: an ingestion pipeline that runs once per document, and a query pipeline that runs on every user question.

Ingestion Pipeline

Query Pipeline

Tech Stack

Every component in this stack is open-source and runs locally. There is no dependency on any paid API or cloud service.

Project Structure

The project is organised as a clean Python package. Each concern (ingestion, indexing, retrieval, LLM, web) lives in its own module.

config.yaml

All tunable parameters live in a single config file. This is the first place to look when adjusting model behaviour or file paths.

ollama:
  base_url: "http://localhost:11434"
  llm_model: "llama3.1:8b"
  embed_model: "nomic-embed-text"

paths:
  data_raw: "./data_raw"
  chroma_db: "./qdrant_db"       # key kept for compatibility
  sqlite_db: "./sqlite/metadata.db"

chunking:
  chunk_size: 512
  chunk_overlap: 64

retrieval:
  top_k: 6
  score_threshold: 0.3

Document Ingestion

The ingestion layer is the most complex part of the pipeline. It handles four fundamentally different file formats, each requiring a different extraction strategy.

PDF Extraction

PDFs come in two flavours: digital (text is directly encoded) and scanned (text exists only as an image). Our extractor handles both:

import fitz  # PyMuPDF
import pytesseract
from PIL import Image

def extract_pdf(filepath):
    doc = fitz.open(filepath)
    pages = []
    for page_num, page in enumerate(doc):
        text = page.get_text("text").strip()
        if not text:
            # Scanned page — render and OCR
            pix = page.get_pixmap(dpi=200)
            img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
            text = pytesseract.image_to_string(img)
        pages.append(ExtractedPage(
            page_number=page_num + 1,
            text=text,
            extraction_method="pymupdf" if text else "ocr"
        ))
    return ExtractedDocument(filename=filepath.name, pages=pages)

Excel Extraction

Excel files require special handling because data is structured in rows and columns, not prose paragraphs. Our extractor finds the real header row (skipping merged cells and branding rows at the top) and converts each sheet to readable text.

Image Extraction

For standalone image files (receipts photographed on a phone, scanned documents), the image extractor applies preprocessing before OCR: converting to greyscale and applying a binary threshold significantly improves Tesseract accuracy on real-world photos.

Deduplication

Before ingesting any file, we compute its SHA-256 hash. If that hash already exists in the SQLite documents table, the file is skipped. This means you can safely re-run the ingestion script on a folder — it will only process new files.

import hashlib

def sha256_file(filepath) -> str:
    h = hashlib.sha256()
    with open(filepath, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()

Chunking Strategy

LLMs have a finite context window. You cannot feed an entire 50-page PDF into a prompt. Chunking is the process of splitting extracted text into smaller pieces that fit within the model's context — while preserving enough surrounding text that each chunk is self-contained and meaningful.

Chunk Size & Overlap

We use RecursiveCharacterTextSplitter from LangChain with these parameters:

• chunk_size = 512 characters — each chunk is roughly one paragraph
• chunk_overlap = 64 characters — consecutive chunks share 64 characters to prevent a sentence from being cut at a chunk boundary and losing context
• Separators — the splitter tries to split on \n\n first, then \n, then spaces, then characters — always preferring natural breaks

  Document text (2000 chars)
  │
  ├── Chunk 1: chars   0 – 511   (512 chars)
  ├── Chunk 2: chars 447 – 959   (512 chars, 64 overlap with chunk 1)
  ├── Chunk 3: chars 895 – 1407  (512 chars, 64 overlap with chunk 2)
  └── Chunk 4: chars 1343 – 1855 (512 chars, 64 overlap with chunk 3)

Chunk Metadata

Every chunk is stored with metadata that travels with it into Qdrant:

{
  "chunk_id":   "abc123-0",         # unique ID
  "doc_id":    "uuid-of-parent-doc",
  "filename":  "Invoice_April.pdf",
  "page_number": 3,
  "chunk_index": 0,
  "doc_type":  "pdf",
  "word_count": 84
}

Embeddings & Vector Storage

What is an Embedding?

An embedding is a list of numbers (a vector) that encodes the semantic meaning of a piece of text. Two pieces of text with similar meaning will have vectors that point in similar directions — their cosine similarity will be close to 1.0.

  "What is the BT bill?"         → [0.12, -0.34, 0.87, ... ]  (768 numbers)
  "BT invoice April £41.17"      → [0.11, -0.31, 0.84, ... ]  ← very similar!
  "The cat sat on the mat"       → [-0.45, 0.67, -0.12, ...]  ← very different

nomic-embed-text

We use nomic-embed-text served by Ollama. It produces 768-dimensional vectors and runs entirely on-device using Metal GPU acceleration. At only 274MB, it is fast enough to embed hundreds of chunks in seconds on a Mac mini M4.

import requests

def embed_text(text: str) -> list[float]:
    resp = requests.post(
        "http://localhost:11434/api/embeddings",
        json={"model": "nomic-embed-text", "prompt": text}
    )
    return resp.json()["embedding"]   # list of 768 floats

Qdrant Vector Store

Vectors are stored in Qdrant running in local file mode — no server process required, just a folder on disk (./qdrant_db/). The collection uses cosine similarity distance.

def query_collection(client, question, n_results=6, filter_metadata=None):
    query_vector = embed_text(question)

    results = client.query_points(
        collection_name="rag_chunks",
        query=query_vector,
        limit=n_results,
        with_payload=True
    )

    hits = []
    for point in results.points:
        hits.append({
            "chunk_id": point.payload["chunk_id"],
            "text":     point.payload["text"],
            "metadata": point.payload,
            "score":    round(point.score, 3)
        })
    return hits

Retrieval

Retrieval is the bridge between the user's question and the LLM's answer. The quality of retrieval directly determines the quality of the final response.

Cosine Similarity

Qdrant compares the query vector against all stored chunk vectors using cosine similarity — a measure of the angle between two vectors in high-dimensional space. A score of 1.0 means identical direction (identical meaning); 0.0 means unrelated.

  Cosine Similarity Score Guide:
  ───────────────────────────────────────────────
  0.85 – 1.00   Very high relevance  ████████████
  0.70 – 0.84   Good relevance       ████████░░░░
  0.50 – 0.69   Moderate             ████░░░░░░░░
  0.30 – 0.49   Weak (often noise)   ██░░░░░░░░░░
  below  0.30   Filter out           ░░░░░░░░░░░░

top_k Parameter

The top_k parameter controls how many chunks are retrieved. Our default is 6. Users can adjust this via a slider in the web UI (range: 3–12). More chunks = more context for the LLM, but also more tokens consumed and potentially more noise.

Metadata Filtering

The UI provides source-type filter pills (All / PDF / Excel / Image). When a filter is active, Qdrant applies a metadata filter so that only chunks from documents of that type are considered in the search.

LLM Chain & Prompting

The LLM chain is where retrieved chunks are assembled into a prompt and sent to llama3.1:8b via Ollama's chat API.

Prompt Structure

The prompt has three parts: a system instruction that defines the LLM's role and output format, the retrieved context chunks with their source labels, and the user's question.

def build_prompt(question: str, hits: list) -> list[dict]:
    context_parts = []
    for i, hit in enumerate(hits, 1):
        fn  = hit["metadata"].get("filename", "?")
        pg  = hit["metadata"].get("page_number", "?")
        txt = hit["text"]
        context_parts.append(f"[Source {i}: {fn}, page {pg}]\n{txt}")

    context = "\n\n".join(context_parts)

    return [
        {"role": "system", "content": (
            "You are a private AI accounting assistant. "
            "Answer ONLY using the provided context. "
            "Always cite your sources as [Source: filename, page X]."
        )},
        {"role": "user", "content":
            f"Context:\n{context}\n\nQuestion: {question}"
        }
    ]

Ollama Chat API Call

def ask(question: str, hits: list) -> str:
    messages = build_prompt(question, hits)

    resp = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model":   "llama3.1:8b",
            "messages": messages,
            "stream":  False,
            "options": {"temperature": 0.1, "num_ctx": 8192}
        },
        timeout=120
    )
    return resp.json()["message"]["content"]

Web UI (Flask)

The web interface is a single-file Flask application (app.py) that serves the AccountIQ dashboard and exposes two API endpoints.

API Endpoints

UI Features

• Workflow sidebar — 6 accounting workflow modes (Ask Documents, VAT Filing, Month-End Recon, etc.)
• Source filter pills — filter retrieval by doc type (PDF / Excel / Image)
• Context chunks slider — adjust top_k from 3 to 12 in real time
• Source chips — each answer shows clickable chips with filename, page, and similarity score
• Chunk modal — click any source chip to see the exact text that grounded the answer
• Query history — sidebar remembers recent queries for quick re-run
• Live KPI counters — document count and vector count polled from /api/stats on load

The 7 Build Stages

The project was built in seven sequential stages. Each stage builds on the previous and can be tested independently.

Key Bugs Fixed Along the Way

Real-world builds always hit unexpected problems. Here are the significant issues resolved during this build — each one is a learning opportunity:

Key Terms Glossary

Where to Go Next

You have built a working local RAG pipeline from scratch. Here are the logical next steps to extend it: