Local RAG Pipeline

A local RAG application built to understand how retrieval-augmented generation works from document ingestion to LLM response.

This project was built as a practical exploration of RAG rather than as a production-ready document search system.

The goal was to understand what happens between uploading a document and asking a local language model a question about its contents.

The pipeline handles:

• PDF text extraction
• Text chunking
• Embedding generation
• Local vector storage
• Semantic similarity search
• Context construction
• Local LLM generation

The idea

The important idea is that the LLM does not need to contain the document's information in its model weights.

Instead, relevant information is retrieved from the document and provided to the model as context.

Document ingestion

When a PDF is uploaded, the backend:

1. Extracts its text using pdfplumber.
2. Splits the text into overlapping chunks.
3. Generates an embedding for each chunk using all-MiniLM-L6-v2.
4. Stores the chunks and embeddings in SQLite.

The embeddings are stored as binary NumPy data alongside the original chunk text.

Retrieval

When a question is submitted, the query is converted into an embedding using the same sentence-transformer model.

The system then calculates cosine similarity between the query embedding and the stored document embeddings.

The three most similar chunks are selected and passed to the LLM as context.

User Question
      │
      ▼
Query Embedding
      │
      ▼
Cosine Similarity
      │
      ▼
Top 3 Chunks
      │
      ▼
Context + Question
      │
      ▼
Ollama
      │
      ▼
Answer

This was the main concept I wanted to understand: retrieval happens before generation.

Local LLM

The generation layer uses Ollama with a local Qwen model.

The retrieved document chunks are inserted into the prompt alongside the user's question, with the model instructed to answer using only the provided context.

The response is streamed back to the frontend using Server-Sent Events.

This keeps the complete pipeline local rather than sending the document or question to an external AI API.

What I learned

The project helped me understand the practical pieces behind a basic RAG system:

• Extracting usable text from documents
• Choosing chunk sizes and overlap
• Generating semantic embeddings
• Comparing embeddings using cosine similarity
• Persisting embeddings alongside source text
• Separating retrieval from generation
• Passing retrieved context to a local LLM
• Streaming LLM responses through FastAPI

The biggest lesson was that RAG is not the same thing as simply asking an LLM about a document. The retrieval step is what gives the model access to information outside its original context.

Current status

Complete — learning-focused RAG implementation

The current implementation supports:

• PDF ingestion
• Text chunking with overlap
• Sentence-transformer embeddings
• SQLite-based embedding storage
• Cosine-similarity retrieval
• Top-3 context selection
• Local Ollama generation
• Streaming responses through FastAPI

The project intentionally uses a simple SQLite-based retrieval approach rather than a dedicated vector database, making the underlying RAG pipeline easier to understand.

Repository