docker: add NVIDIA/AMD GPU overlays via COMPOSE_FILE (#254)

Opt-in overlays under docker/ that pass the host GPU into the odysseus
container. Pick one in .env:

  COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
  COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml

Non-GPU users are unaffected (no default merge). README now points at
the overlays instead of the old ad-hoc `gpus: all` suggestion.

Each overlay header notes that it only exposes the GPU devices — the
slim image still needs vLLM / llama-cpp-python / etc. installed via
Cookbook -> Dependencies before models can serve on GPU.

Tested on Arch + Docker 29.5.1 + RTX 4090:
  docker compose exec odysseus nvidia-smi -L
  GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...)
Cookbook hardware scan reports the 24 GB GPU and recommends GPU-fit
models. `docker compose config` validates cleanly for all three
COMPOSE_FILE variants (base, +nvidia, +amd).

Builds on the structure proposed in #91 by @krllus with the path /
docs fixes from the review on that PR.

Closes #163.

Co-authored-by: krllus <krllus@users.noreply.github.com>
This commit is contained in:
Sirsyorrz
2026-06-01 15:00:09 +10:00
committed by GitHub
parent 2537b80f88
commit 4c0aadbb5e
4 changed files with 78 additions and 3 deletions

18
docker/gpu.amd.yml Normal file
View File

@@ -0,0 +1,18 @@
# AMD ROCm GPU overlay. Enable by setting COMPOSE_FILE in .env:
# COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
#
# Requires ROCm drivers on the host (kfd + DRI devices). The host user
# running Docker must be in the `video` and `render` groups.
#
# This overlay only passes the host GPU through to the container.
# The slim Odysseus image does not bundle ROCm userspace or inference
# engines — install ROCm-compatible builds of vLLM / llama-cpp-python
# via Cookbook -> Dependencies (or pip) before serving GPU models.
services:
odysseus:
devices:
- /dev/kfd
- /dev/dri
group_add:
- video
- render

29
docker/gpu.nvidia.yml Normal file
View File

@@ -0,0 +1,29 @@
# NVIDIA GPU overlay. Enable by setting COMPOSE_FILE in .env:
# COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
#
# Requires the NVIDIA Container Toolkit on the host.
# Arch: sudo pacman -S nvidia-container-toolkit
# Debian: sudo apt install nvidia-container-toolkit
# Fedora: sudo dnf install nvidia-container-toolkit
# Then:
# sudo nvidia-ctk runtime configure --runtime=docker
# sudo systemctl restart docker
# Verify with:
# docker info | grep -i nvidia
#
# This overlay only passes the host GPU through to the container.
# The slim Odysseus image does not bundle CUDA userspace or inference
# engines — install vLLM / llama-cpp-python / SGLang via
# Cookbook -> Dependencies (or pip) before serving GPU models.
services:
odysseus:
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]