docker: add NVIDIA/AMD GPU overlays via COMPOSE_FILE (#254)
Opt-in overlays under docker/ that pass the host GPU into the odysseus container. Pick one in .env: COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml Non-GPU users are unaffected (no default merge). README now points at the overlays instead of the old ad-hoc `gpus: all` suggestion. Each overlay header notes that it only exposes the GPU devices — the slim image still needs vLLM / llama-cpp-python / etc. installed via Cookbook -> Dependencies before models can serve on GPU. Tested on Arch + Docker 29.5.1 + RTX 4090: docker compose exec odysseus nvidia-smi -L GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-...) Cookbook hardware scan reports the 24 GB GPU and recommends GPU-fit models. `docker compose config` validates cleanly for all three COMPOSE_FILE variants (base, +nvidia, +amd). Builds on the structure proposed in #91 by @krllus with the path / docs fixes from the review on that PR. Closes #163. Co-authored-by: krllus <krllus@users.noreply.github.com>
This commit is contained in:
18
docker/gpu.amd.yml
Normal file
18
docker/gpu.amd.yml
Normal file
@@ -0,0 +1,18 @@
|
||||
# AMD ROCm GPU overlay. Enable by setting COMPOSE_FILE in .env:
|
||||
# COMPOSE_FILE=docker-compose.yml:docker/gpu.amd.yml
|
||||
#
|
||||
# Requires ROCm drivers on the host (kfd + DRI devices). The host user
|
||||
# running Docker must be in the `video` and `render` groups.
|
||||
#
|
||||
# This overlay only passes the host GPU through to the container.
|
||||
# The slim Odysseus image does not bundle ROCm userspace or inference
|
||||
# engines — install ROCm-compatible builds of vLLM / llama-cpp-python
|
||||
# via Cookbook -> Dependencies (or pip) before serving GPU models.
|
||||
services:
|
||||
odysseus:
|
||||
devices:
|
||||
- /dev/kfd
|
||||
- /dev/dri
|
||||
group_add:
|
||||
- video
|
||||
- render
|
||||
29
docker/gpu.nvidia.yml
Normal file
29
docker/gpu.nvidia.yml
Normal file
@@ -0,0 +1,29 @@
|
||||
# NVIDIA GPU overlay. Enable by setting COMPOSE_FILE in .env:
|
||||
# COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
|
||||
#
|
||||
# Requires the NVIDIA Container Toolkit on the host.
|
||||
# Arch: sudo pacman -S nvidia-container-toolkit
|
||||
# Debian: sudo apt install nvidia-container-toolkit
|
||||
# Fedora: sudo dnf install nvidia-container-toolkit
|
||||
# Then:
|
||||
# sudo nvidia-ctk runtime configure --runtime=docker
|
||||
# sudo systemctl restart docker
|
||||
# Verify with:
|
||||
# docker info | grep -i nvidia
|
||||
#
|
||||
# This overlay only passes the host GPU through to the container.
|
||||
# The slim Odysseus image does not bundle CUDA userspace or inference
|
||||
# engines — install vLLM / llama-cpp-python / SGLang via
|
||||
# Cookbook -> Dependencies (or pip) before serving GPU models.
|
||||
services:
|
||||
odysseus:
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
Reference in New Issue
Block a user