00320972dc
Two bugs caused GPU inference to silently fall back to CPU inside the Odysseus Docker container even when the GPU was correctly passed through. ## entrypoint.sh — CUDA_HOME detection only covered CUDA 13.x wheels The nvcc glob only searched vidia/cu13, which matches the vidia-nvcc-cu13 pip wheel layout. CUDA 12.x wheels install nvcc to vidia/cuda_nvcc/bin/nvcc (nvidia-cuda-nvcc-cu12) or vidia/cu12 (nvidia-nvcc-cu12) — completely different paths. The glob found nothing, so CUDA_HOME was never set. Worse, VLLM_USE_FLASHINFER_SAMPLER=0 was inside the same if-block, so it was never set either. vLLM then tried to JIT-compile the FlashInfer sampler at startup, failed with 'Could not find nvcc', and crashed — even though the GPU was fully visible to the container. Fix: expand the search to also check nvidia/cu12 and nvidia/cuda_nvcc. Move VLLM_USE_FLASHINFER_SAMPLER=0 to an unconditional export after the loop (it is sampler-only, no impact on the attention path, and the correct setting for any container where CUDA headers may be incomplete). ## cookbook_routes.py — llama.cpp Linux source build silently fell back to CPU The cmake invocation was: cmake -B build -DGGML_CUDA=ON 2>/dev/null || cmake -B build 2>/dev/null suppressed all configure errors. When nvcc is absent (the slim base image has no CUDA toolkit — intentional), cmake fails silently, then the || fallback re-runs without -DGGML_CUDA=ON. A CPU-only binary is produced with no warning. Additionally, a stale CMakeCache.txt from the failed CUDA attempt was reused (no rm -rf build), poisoning the next configure run. The macOS branch already did rm -rf build for exactly this reason; the Linux branch did not. Fix: before cmake, detect pip-installed nvcc across the same three path patterns as entrypoint.sh and expose it via CUDA_HOME/PATH. If nvcc is found, run a clean CUDA build with full error visibility. If not, fall back to a CPU build with an explicit warning telling the user how to get a GPU build (install vLLM via Cookbook -> Dependencies, which brings the CUDA wheels including nvcc, then re-launch). ## .env.example — document Windows COMPOSE_FILE separator Added a comment showing the semicolon separator required on Windows Docker Desktop alongside the existing colon-separator (Linux) example.
83 lines
3.7 KiB
Bash
83 lines
3.7 KiB
Bash
#!/bin/sh
|
|
# Entrypoint that fixes the #1 self-host footgun: a Docker container
|
|
# that runs as root writes root-owned files into bind-mounted host
|
|
# volumes, and the host user (or a non-root service user) then can't
|
|
# update them — silently breaking skill extraction, prefs saves, mail
|
|
# attachments, etc.
|
|
#
|
|
# Standard PUID/PGID pattern: pick the UID/GID we should drop to,
|
|
# chown the writable bind-mounts so existing root-owned content gets
|
|
# repaired on every start (idempotent), then exec the real command
|
|
# as that user via gosu.
|
|
set -e
|
|
|
|
PUID="${PUID:-1000}"
|
|
PGID="${PGID:-1000}"
|
|
|
|
# Reuse an existing matching group/user if the host's UID/GID already
|
|
# corresponds to one in /etc/passwd (e.g. when the image is rebuilt
|
|
# and "odysseus" already exists at the same id). Otherwise create.
|
|
if ! getent group "$PGID" >/dev/null 2>&1; then
|
|
groupadd -g "$PGID" odysseus
|
|
fi
|
|
if ! getent passwd "$PUID" >/dev/null 2>&1; then
|
|
useradd -u "$PUID" -g "$PGID" -M -s /bin/sh -d /app odysseus
|
|
fi
|
|
|
|
# Repair ownership on every writable path the app touches at runtime.
|
|
#
|
|
# Bind-mounted dirs (/app/data, /app/logs) are the obvious ones, but
|
|
# the app ALSO writes inside the image's own source tree at runtime:
|
|
# - services/cache/{search,content}/* (search cache LRU)
|
|
# - services/search_analytics.json
|
|
# - services/search_engine_error.log
|
|
# - services/tts cache, etc.
|
|
# These dirs were created as root during `docker build`, so dropping
|
|
# to PUID:PGID would otherwise crash on the first import that tries
|
|
# to mkdir them. Chown the whole /app tree — fast (<1s on this size)
|
|
# and idempotent via the `-not -uid` filter so we only touch files
|
|
# that need fixing.
|
|
for dir in /app /app/data /app/logs; do
|
|
if [ -d "$dir" ]; then
|
|
# `find ... -not -uid` keeps this O(touched-files), not
|
|
# O(everything), so terabyte-sized maildirs don't slow startup.
|
|
find "$dir" -not -uid "$PUID" -print0 2>/dev/null \
|
|
| xargs -0 -r chown "$PUID:$PGID" 2>/dev/null || true
|
|
fi
|
|
done
|
|
|
|
# Cookbook installs vllm/etc. via `pip install --user`, which pulls
|
|
# nvidia-cuda-* wheels into /app/.local but does not set CUDA_HOME or
|
|
# symlink /usr/local/cuda. vllm 0.22+ then crashes during engine init
|
|
# when FlashInfer tries to JIT a sampler kernel ("Could not find nvcc",
|
|
# then "CUDA compiler and toolkit headers are incompatible" on the
|
|
# mixed cuda-nvcc 13.3 / cuda-runtime 13.0 wheel combo).
|
|
#
|
|
# Auto-set CUDA_HOME if a pip-installed nvcc is present, and disable the
|
|
# FlashInfer JIT sampler — sampler only, no impact on attention path.
|
|
# No-op when vllm isn't installed.
|
|
#
|
|
# Checked layouts (all are real pip-wheel install paths):
|
|
# nvidia/cu13 — nvidia-nvcc-cu13 (CUDA 13.x wheel style)
|
|
# nvidia/cu12 — nvidia-nvcc-cu12 (CUDA 12.x wheel style)
|
|
# nvidia/cuda_nvcc — nvidia-cuda-nvcc-cu12 (older cu12 sub-package style)
|
|
for cu in \
|
|
/app/.local/lib/python*/site-packages/nvidia/cu13 \
|
|
/app/.local/lib/python*/site-packages/nvidia/cu12 \
|
|
/app/.local/lib/python*/site-packages/nvidia/cuda_nvcc; do
|
|
if [ -x "$cu/bin/nvcc" ]; then
|
|
export CUDA_HOME="$cu"
|
|
break
|
|
fi
|
|
done
|
|
# Disable the FlashInfer JIT sampler unconditionally — it is sampler-only
|
|
# and has no impact on the attention path, but requires nvcc + matching
|
|
# CUDA headers at startup. Without this, vLLM crashes with "Could not find
|
|
# nvcc" even when the GPU itself is fully visible to the container.
|
|
export VLLM_USE_FLASHINFER_SAMPLER="${VLLM_USE_FLASHINFER_SAMPLER:-0}"
|
|
|
|
# Drop root and run the actual app. `gosu` is preferred over `su` /
|
|
# `sudo` because it cleans up the process tree (no extra shell layer)
|
|
# so signals (SIGTERM from `docker stop`) reach uvicorn directly.
|
|
exec gosu "$PUID:$PGID" "$@"
|