Setting Up Ollama for Local Homelab AI Inference

Welcome to By Design Won't Fix, a daily hands-on build guide where we pick one piece of self-hosted infrastructure or homelab tooling, stand it up from scratch, and wire it into a working system. To kick off the series, we are setting up Ollama in Docker on a local Linux GPU server, with a practical twist I learned the hard way: running two isolated container instances side by side so scheduled background agents do not constantly evict the model you are actively using in your IDE.

When you run a single Ollama container on port 11434 and point your coding assistant, your home automation scripts, and your cron jobs at the same endpoint, the first background job that requests a different model weight will unload your active coding model from VRAM. Splitting your deployment into a primary ollama-hub container on port 11434 and a secondary ollama-worker container on port 11435 gives you clean workload isolation while keeping everything reproducible in a single Compose file.

Step 1: Verify NVIDIA Container Toolkit Access

Before creating the Compose stack on your GPU host (gpu-node-01), make sure the NVIDIA Container Toolkit is installed and Docker can see your GPU hardware. Run a quick throwaway check against nvidia-smi:

sudo mkdir -p /opt/homelab/ollama/{hub-data,worker-data}
sudo chown -R 1000:1000 /opt/homelab/ollama
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If you see your GPU table printed cleanly, Docker is ready to pass device reservations into the Ollama runtime.

Step 2: Create the Dual-Instance Docker Compose File

Create /opt/homelab/ollama/docker-compose.yml to define both the primary ollama-hub service on port 11434 and the isolated ollama-worker service on host port 11435. Both containers set OLLAMA_HOST=0.0.0.0:11434 internally so they accept connections from other hosts and containers on your local subnet:

services:
  ollama-hub:
    image: ollama/ollama:latest
    container_name: ollama-hub
    restart: unless-stopped
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=2
    volumes:
      - /opt/homelab/ollama/hub-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  ollama-worker:
    image: ollama/ollama:latest
    container_name: ollama-worker
    restart: unless-stopped
    ports:
      - "11435:11434"
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_KEEP_ALIVE=15m
      - OLLAMA_NUM_PARALLEL=1
    volumes:
      - /opt/homelab/ollama/worker-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Step 3: Start the Stack and Pull Your Initial Models

Bring both containers online and pull a fast coding/instruction model into the primary hub and a lightweight utility model into the worker instance:

cd /opt/homelab/ollama
docker compose up -d
docker exec -it ollama-hub ollama pull qwen2.5-coder:14b
docker exec -it ollama-worker ollama pull llama3.2:3b

Because /root/.ollama is mounted to persistent host directories under /opt/homelab/ollama/, upgrading the container image (docker compose pull && docker compose up -d) will never force you to re-download multi-gigabyte GGUF blobs.

Step 4: Test Both Endpoints Over HTTP

Verify that both ports respond independently and list only the models assigned to their respective volume mounts:

curl -s http://localhost:11434/api/tags | jq '.models[].name'
curl -s http://localhost:11435/api/tags | jq '.models[].name'

You can also hit the built-in OpenAI-compatible chat completions endpoint at http://gpu-node-01.lan:11434/v1/chat/completions from any client on your network using api_key="ollama".

Hardware & Gear Used in This Guide

• NVIDIA RTX 24GB+ Homelab GPU Cards (Amazon Search): Having at least 16 GB to 24 GB of VRAM lets you keep a 14B coding model loaded in ollama-hub alongside a 3B worker model in ollama-worker.
• Samsung 990 PRO 2TB PCIe 4.0 NVMe M.2 SSD (Amazon Search): High-throughput NVMe storage cuts cold model weight loading times down to a few seconds when switching GGUF blobs.
• 2.5GbE Managed Homelab Switch (Amazon Search): Keeps LAN latency negligible when streaming tokens between your Docker host, Proxmox nodes, and workstation.

Disclosure: Some hardware and infrastructure links above are affiliate or referral links that support By Design Won't Fix at no extra cost to you.

What You Can Do Next

• Place a LiteLLM proxy container in front of ports 11434 and 11435 so all of your homelab tools talk to a single unified OpenAI gateway with automatic fallback.
• Connect Home Assistant's Local Conversation integration to the ollama-worker port (11435) for low-latency voice intent parsing without disturbing your primary coding model.
• Mount a shared read-only models/blobs directory between both containers if you want them to share underlying weight files on disk while keeping separate runtime queues.

Later,
     SteveO

☕ Fuel the Next Homelab Build

If this guide saved you an hour of debugging YAML or hunting down kernel flags, consider buying me a coffee to help fund the next round of lab hardware and tutorials.

☕ Buy Me a Coffee