By Design Won't Fix
☕ Buy Me a Coffee

Setting Up Hermes Agent Locally with a LiteLLM Router and Free Cloud Fallbacks

November 06, 2026
Autonomous Hermes Agent container paired with a LiteLLM routing switchboard across local GPUs and cloud fallback models

Earlier this week we set up local Ollama and vLLM inference servers; today we are deploying the NousResearch Hermes Agent container (nousresearch/hermes-agent:latest) paired with a LiteLLM router so our autonomous agent can execute cron jobs, maintain a Markdown knowledge vault, and fail over gracefully between local GPUs and cloud APIs.

Running an always-on agent against a local Git-backed Markdown vault introduces two classic operational traps: model endpoint timeouts when your local GPU is busy, and PermissionError crashes when a container writes root-owned files into a volume shared with your host user. We will solve both in a single Compose stack.

Prerequisites

Before starting this build, you need a Linux host with Docker and Docker Compose installed, running under a standard non-root user account (UID 1000). Next, you will need a Git repository to back your Markdown knowledge vault and your container configurations; you could host your own or use a free account on GitHub or GitLab (I self-host GitLab for the unlimited pipeline quota). Finally, you need at least one local inference endpoint (such as the Ollama or vLLM servers from earlier this week) and an optional OpenRouter or cloud API key for fallback routing.

  • Linux host with Docker Engine and Docker Compose installed, plus a standard non-root user account (UID 1000).
  • Git repository for your Obsidian Markdown vault and Compose files (GitHub, GitLab, or self-hosted GitLab).
  • Local LLM inference server on your LAN (such as Ollama on port 11434 or vLLM on port 8000).
  • OpenRouter API key (or another OpenAI-compatible cloud API key) for automatic fallback routing.

Step 1: Create Non-Root Host Directories Owned by UID 1000

On your container host (ai-host-01), create the persistent data directories for Hermes Agent and LiteLLM, and explicitly ensure every file is owned by your standard user (UID 1000 / GID 1000). If you mount a Git-backed Markdown vault into the container, never run docker exec commands without passing -u 1000:1000 (or -u hermes), or Git pulls will leave root-owned files behind that break scheduled agent jobs:

mkdir -p /path/to/git/hermes-agent/{config,vault,logs,litellm}
sudo chown -R 1000:1000 /path/to/git/hermes-agent

Step 2: Configure the LiteLLM Router with Local-First Fallbacks

Create /path/to/git/hermes-agent/litellm/config.yaml to expose a single unified model alias (hermes-default) on port 4000. LiteLLM will route requests to your local vLLM or Ollama instance first and automatically fall back to a cloud provider if the local GPU queue is saturated:

model_list:
  - model_name: hermes-local-primary
    litellm_params:
      model: openai/qwen2.5-coder-14b
      api_base: http://gpu-node-01.lan:8000/v1
      api_key: os.environ/VLLM_API_KEY
      rpm: 60

  - model_name: hermes-local-ollama
    litellm_params:
      model: ollama/qwen2.5-coder:14b
      api_base: http://gpu-node-01.lan:11434

  - model_name: hermes-cloud-fallback
    litellm_params:
      model: openrouter/qwen/qwen-2.5-coder-32b-instruct:free
      api_key: os.environ/OPENROUTER_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 90
  fallbacks:
    - {"hermes-local-primary": ["hermes-local-ollama", "hermes-cloud-fallback"]}

Step 3: Deploy Hermes Agent and LiteLLM with Docker Compose

Create /path/to/git/hermes-agent/docker-compose.yml to launch both services on a shared bridge network. Note how hermes-agent mounts /path/to/git/hermes-agent to /app/data and points its OpenAI base URL directly at the internal litellm:4000 service:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "4000:4000"
    env_file:
      - .env
    volumes:
      - /path/to/git/hermes-agent/litellm/config.yaml:/app/config.yaml:ro
    command: ["--config", "/app/config.yaml", "--port", "4000"]

  hermes-agent:
    image: nousresearch/hermes-agent:latest
    container_name: hermes-agent
    user: "1000:1000"
    restart: unless-stopped
    depends_on:
      - litellm
    ports:
      - "8642:8642"
      - "9119:9119"
    env_file:
      - .env
    environment:
      - HERMES_HOME=/app/data
      - OPENAI_BASE_URL=http://litellm:4000/v1
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      - DEFAULT_MODEL=hermes-local-primary
    volumes:
      - /path/to/git/hermes-agent:/app/data

Step 4: Start the Stack and Verify File Ownership Hygiene

Bring both containers up and verify that the LiteLLM health check passes and the mounted vault contains zero non-UID-1000 files:

cd /path/to/git/hermes-agent
docker compose up -d
curl -s http://localhost:4000/health/readiness | jq .
find /path/to/git/hermes-agent/vault ! -uid 1000

The find command should print nothing. Whenever you sync the vault from inside the container, always run docker exec -u 1000:1000 hermes-agent git -C /app/data/vault pull --ff-only origin main to preserve clean ownership.

Hardware & Gear Used in This Guide

Disclosure: Some hardware and infrastructure links above are affiliate or referral links that support By Design Won't Fix at no extra cost to you.

What You Can Do Next

  • Configure scheduled morning cron jobs inside Hermes Agent to audit homelab container health, triage unread emails, and write structured status summaries into your Daily Note.
  • Add an AGENTS.md dispatcher file at the root of /app/data/vault to route vault tasks across specialized skills (architect, scribe, librarian, and seeker).
  • Expose port 8642 internally to your local dashboard or Telegram bot bridge so you can message your agent from your phone over a private VPN.

Later,
     SteveO

☕ Fuel the Next Homelab Build
If this guide saved you an hour of debugging YAML or hunting down kernel flags, consider buying me a coffee to help fund the next round of lab hardware and tutorials.
☕ Buy Me a Coffee