Building a Private RAG Pipeline with Qdrant, Ollama, and Docker

Running large language models locally is powerful, but out-of-the-box models lack context about your private codebase, documentation, or personal notes. To give your local LLM domain-specific knowledge without leaking data to third-party APIs, you need a Retrieval-Augmented Generation (RAG) pipeline.

In this guide, we will break down how to wire together Ollama, Qdrant, and Open WebUI inside a unified Docker network so you can chat with your local documents securely and efficiently.

The Local RAG Architecture

A standard local RAG stack relies on three core services working in tandem:

1. The Inference Engine (Ollama): Handles model execution and text generation.
2. The Embedding Model & Vector DB (Qdrant): Converts raw text into high-dimensional vectors and performs lightning-fast semantic similarity searches.
3. The User Interface (Open WebUI): Provides a ChatGPT-style frontend that orchestrates document chunking, embedding generation, and prompt injection.

Manually aligning port mappings, environment variables, and persistent volumes for all three can quickly become a debugging headache. (If you want a zero-config, production-hardened version of this exact architecture, you can check out our Pro Docker Stack).

Step-by-Step Implementation

1. Define Your Docker Compose Services

Create a docker-compose.yml file to orchestrate Ollama and Qdrant with proper persistent volumes:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: local_ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

  qdrant:
    image: qdrant/qdrant:latest
    container_name: local_qdrant
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - qdrant_data:/qdrant/storage
    restart: unless-stopped

volumes:
  ollama_data:
  qdrant_data: