Setting Up vLLM for High-Throughput OpenAI-Compatible Serving

Setting Up vLLM for High-Throughput OpenAI-Compatible Serving

After standing up Ollama for quick model switching, today we are deploying vLLM (vllm/vllm-openai:latest) in Docker on port 8000 to serve a dedicated OpenAI-compatible API built for concurrent multi-agent workloads.

Ollama is tough to beat when you want to swap models on the fly, but as soon as you have three or four autonomous agents firing parallel tool-calling prompts at the same time, request queuing starts to hurt. vLLM uses PagedAttention and continuous batching to process concurrent requests simultaneously in GPU memory, turning a single homelab GPU into a genuine multi-tenant inference server.

Prerequisites

Before starting this build, you need a Linux GPU host with Docker, Docker Compose, and the NVIDIA Container Toolkit installed so containers can access your CUDA devices. Next, you will need a Git repository to store your Compose configuration and environment template; you could host your own or use a free account on GitHub or GitLab (I self-host GitLab for the unlimited pipeline quota). Finally, you will need a free Hugging Face account and a read-only access token (hf_...) so the container can download quantized AWQ or GPTQ model weights into your persistent cache volume.

• Linux GPU server (16 GB to 24 GB VRAM recommended) with NVIDIA drivers and the NVIDIA Container Toolkit installed.
• Docker Engine and Docker Compose installed on the GPU host.
• Git repository for tracking your Compose stack (GitHub, GitLab, or self-hosted GitLab).
• Hugging Face account with a read-only access token (HF_TOKEN) and 40 GB+ of free NVMe storage for cached weights.

Step 1: Prepare the Hugging Face Cache and Environment File

On your GPU server (gpu-node-01), create a dedicated directory for the vLLM Compose stack and a persistent cache directory so Hugging Face model weights are downloaded once and reused across container restarts:

sudo mkdir -p /path/to/git/vllm /path/to/cache/huggingface
sudo chown -R 1000:1000 /path/to/git/vllm /path/to/cache/huggingface

Next, create /path/to/git/vllm/.env to store your Hugging Face read token (needed if you pull gated models) and a local bearer token for your internal clients:

HF_TOKEN=hf_your_read_only_token_here
VLLM_API_KEY=local-vllm-service-key
VLLM_MODEL=Qwen/Qwen2.5-Coder-14B-Instruct-AWQ

Step 2: Write the vLLM Docker Compose Configuration

Create /path/to/git/vllm/docker-compose.yml. Setting ipc: host (or a generous shm_size) is critical for PyTorch shared memory communication, and capping --gpu-memory-utilization at 0.85 leaves headroom on the card if you run a small embedding or reranker process alongside vLLM:

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    ipc: host
    ports:
      - "8000:8000"
    env_file:
      - .env
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - /path/to/cache/huggingface:/root/.cache/huggingface
    command: >
      --model ${VLLM_MODEL}
      --served-model-name qwen2.5-coder-14b
      --quantization awq
      --dtype auto
      --max-model-len 16384
      --gpu-memory-utilization 0.85
      --enable-auto-tool-choice
      --tool-call-parser hermes
      --api-key ${VLLM_API_KEY}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Step 3: Launch the Server and Watch the KV-Cache Warmup

Start the container and tail the logs while vLLM downloads the quantized safetensors weights, profiles available VRAM blocks, and compiles the CUDA graphs:

cd /path/to/git/vllm
docker compose up -d
docker compose logs -f vllm

Once you see Uvicorn running on http://0.0.0.0:8000, the server is ready to accept concurrent requests.

Step 4: Verify the OpenAI-Compatible Endpoint

Send a test completion request against /v1/chat/completions using the bearer key from your .env file to confirm the model responds cleanly:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer local-vllm-service-key" \
  -d '{
    "model": "qwen2.5-coder-14b",
    "messages": [
      {"role": "system", "content": "Reply in one concise sentence."},
      {"role": "user", "content": "Explain why continuous batching improves multi-agent throughput."}
    ],
    "temperature": 0.2
  }' | jq -r '.choices[0].message.content'

Hardware & Gear Used in This Guide

• NVIDIA 24GB VRAM GPU for AWQ Quantized Serving (Amazon Search): A 24 GB card comfortably fits a 14B 4-bit AWQ model with a 16K context window and plenty of PagedAttention KV-cache blocks for parallel agents.
• 64GB DDR5 ECC / High-Speed Server Memory Kit (Amazon Search): Prevents host RAM bottlenecks while PyTorch initializes safetensors weights and shared memory (ipc: host).
• Pure Sine Wave 1500VA Rack/Tower UPS Battery Backup (Amazon Search): Protects your GPU server during sustained multi-agent inference loads.

Disclosure: Some hardware and infrastructure links above are affiliate or referral links that support By Design Won't Fix at no extra cost to you.

What You Can Do Next

• Scrape http://gpu-node-01.lan:8000/metrics with Prometheus and Grafana to track real-time KV-cache usage, time-to-first-token (TTFT), and tokens-per-second under load.
• Add --enable-prefix-caching to the command block so agents that share a long system prompt skip recomputing the initial KV blocks on every turn.
• Register your port 8000 vLLM instance as the primary high-concurrency backend in a LiteLLM router or autonomous agent framework.

Later,
     SteveO

☕ Fuel the Next Homelab Build

If this guide saved you an hour of debugging YAML or hunting down kernel flags, consider buying me a coffee to help fund the next round of lab hardware and tutorials.

☕ Buy Me a Coffee