diff --git a/README.md b/README.md index 6310bef..9c5b5f4 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,10 @@ image build. Open `http://localhost:7000` after the containers are healthy. If port `7000` is already taken, set `APP_PORT=7001` (or another free port) in `.env`, recreate the container, and open `http://localhost:7001`. +> **On Apple Silicon, Docker can't use the Metal GPU** (it runs a Linux VM), so +> Cookbook will serve models on the CPU only. For GPU-accelerated Cookbook, +> run the app natively — see [Apple Silicon](#apple-silicon-m-series). + Cookbook remote servers use an Odysseus-owned SSH key from `./data/ssh` inside Docker. In **Cookbook -> Settings -> Servers**, generate/copy the public key and add it to the remote server's `~/.ssh/authorized_keys`. @@ -111,8 +115,12 @@ The Cookbook model catalog check should print a non-zero count. If it prints `0`, rebuild the Odysseus image with `docker compose build --no-cache odysseus`. ### Option 2: Manual install — Linux / macOS -**Requirements:** Python 3.11+. On Linux/Termux, Cookbook also requires `tmux` -for background model downloads and serves. +**Requirements:** Python 3.11+. Cookbook also requires `tmux` for background +model downloads and serves. + +> **On macOS (Apple Silicon)?** Skip the manual steps below — run +> `./start-macos.sh` for a one-command setup. See +> [Apple Silicon](#apple-silicon-m-series). Install system packages first: ```bash @@ -124,19 +132,81 @@ sudo pacman -S tmux # Fedora sudo dnf install tmux + +# macOS (Homebrew). macOS ships no recent Python by default — install 3.11+ +# (skip the python line if you already have Python 3.11 or newer): +brew install python@3.11 tmux ``` Then install Odysseus: ```bash git clone https://github.com/pewdiepie-archdaemon/odysseus.git cd odysseus -python3 -m venv venv +python3 -m venv venv # on macOS use: python3.11 -m venv venv source venv/bin/activate pip install -r requirements.txt python setup.py # creates data dirs and prints an initial admin password python -m uvicorn app:app --host 0.0.0.0 --port 7000 ``` +#### Apple Silicon (M-series) + +> **On a Mac, run Odysseus natively (not in Docker) so Cookbook can use the +> Metal GPU.** Cookbook serves models on whatever machine Odysseus runs on, and +> Docker on macOS is a Linux VM with **no access to the GPU** — in a container +> your Mac looks like a CPU-only Linux box. + +**Quick start — one command.** From a fresh clone: +```bash +git clone https://github.com/pewdiepie-archdaemon/odysseus.git +cd odysseus +./start-macos.sh +``` +That installs what's needed via Homebrew (Python 3.11+, `tmux`, and a prebuilt +Metal `llama-server`), sets everything up, and launches Odysseus at +**http://127.0.0.1:7860**. Log in with the admin password it prints, open +**Cookbook**, and it detects your GPU (`backend: metal`) and recommends GGUF +models that fit your Mac. (MLX models aren't supported on macOS and are hidden — +see below.) Re-run `./start-macos.sh` any time to start it again (use another +port with `ODYSSEUS_PORT=7900 ./start-macos.sh`). + +**Prefer a clickable app?** After your first `./start-macos.sh`, build a +launcher `Odysseus.app` (+ a drag-to-Applications `.dmg`) that starts the server +and opens the UI in its own window: +```bash +./build-macos-app.sh # → dist/Odysseus.app and dist/Odysseus.dmg +``` + +
+What start-macos.sh does, serving engines, and manual steps + +`start-macos.sh` is just the manual steps wrapped up: Homebrew deps → a Python +`venv` → `pip install -r requirements.txt` → `python setup.py` → `uvicorn` on a +non-AirPlay port. Run them by hand if you prefer (the Linux steps above, but use +`python3.11 -m venv` and `--port 7860`). + +**Serving engines on Metal** — Cookbook only recommends models it can serve here: +- **llama.cpp** — `brew install llama.cpp` (done by `start-macos.sh`) provides a + prebuilt Metal `llama-server`, no compile. Without it, Cookbook builds it from + source on first serve, which needs `cmake` + Xcode Command Line Tools + (`brew install cmake && xcode-select --install`). +- **Ollama** — `brew install ollama` is another simple Metal-accelerated option. +- vLLM/SGLang are CUDA/ROCm-only and do **not** run on macOS. + +**MLX models are not supported on Apple Silicon.** Odysseus serves models via +llama.cpp/Ollama (GGUF) and vLLM/SGLang (CUDA) — it has no MLX (`mlx_lm`) +runtime. So MLX-only models can't be served on a Mac and are deliberately +**hidden** from Cookbook's recommendations there; pick a GGUF build instead. + +**Port 7000 & AirPlay** — macOS AirPlay Receiver holds ports 7000/5000, so +`start-macos.sh` defaults to **7860**. To use 7000, turn AirPlay Receiver off in +System Settings → General → AirDrop & Handoff. + +**Build prerequisites baked in** — the `.app` wraps this repo's `venv` (it +doesn't bundle Python), so the path is fixed at build time — rebuild if you move +the repo. +
+ ### Option 3: Manual install — Windows (PowerShell) Windows support is not actively tested. Use it with caution; Docker on Linux or a Linux/macOS manual install is the safer path for now. diff --git a/build-macos-app.sh b/build-macos-app.sh new file mode 100755 index 0000000..7413181 --- /dev/null +++ b/build-macos-app.sh @@ -0,0 +1,169 @@ +#!/bin/bash +# Build a downloadable macOS launcher app + .dmg for Odysseus. +# +# ./build-macos-app.sh +# +# Produces: +# dist/Odysseus.app — double-click: starts the local server (using this +# repo's venv) and opens the UI in an app-style window. +# dist/Odysseus.dmg — drag-to-Applications disk image (the downloadable). +# +# This is a *launcher* wrapper: it drives the venv we set up in this repo, it +# does not bundle Python. The install path is baked into the app at build time, +# so rebuild if you move the repo. Override the port with ODYSSEUS_PORT. +set -e + +REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +APP_NAME="Odysseus" +INSTALL_DIR="$REPO_DIR" +PORT="${ODYSSEUS_PORT:-7860}" +DIST="$REPO_DIR/dist" +APP="$DIST/$APP_NAME.app" + +echo "Building $APP_NAME.app" +echo " install dir: $INSTALL_DIR" +echo " port: $PORT" + +rm -rf "$APP" +mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources" + +# ── Icon (best effort) — center-crop docs/odysseus.jpg to a square .icns ── +if [ -f "$REPO_DIR/docs/odysseus.jpg" ] && command -v sips >/dev/null 2>&1; then + TMPIMG="$(mktemp -d)" + # Center-crop to a square, scale to 512 (sips' icns encoder caps at 512), and + # let sips emit the .icns directly — more robust across macOS versions than + # building an .iconset by hand. + sips -c 720 720 "$REPO_DIR/docs/odysseus.jpg" --out "$TMPIMG/sq.png" >/dev/null 2>&1 || cp "$REPO_DIR/docs/odysseus.jpg" "$TMPIMG/sq.png" + sips -z 512 512 "$TMPIMG/sq.png" --out "$TMPIMG/icon.png" >/dev/null 2>&1 + if sips -s format icns "$TMPIMG/icon.png" --out "$APP/Contents/Resources/odysseus.icns" >/dev/null 2>&1; then + echo " icon: odysseus.icns" + else + echo " icon: (skipped — conversion failed)" + fi + rm -rf "$TMPIMG" +else + echo " icon: (skipped — no docs/odysseus.jpg)" +fi + +# ── Info.plist ── +cat > "$APP/Contents/Info.plist" < + + + + CFBundleName $APP_NAME + CFBundleDisplayName $APP_NAME + CFBundleIdentifier com.odysseus.launcher + CFBundleVersion 1.0 + CFBundleShortVersionString1.0 + CFBundlePackageType APPL + CFBundleExecutable $APP_NAME + CFBundleIconFile odysseus + LSMinimumSystemVersion 11.0 + NSHighResolutionCapable + LSUIElement + + +PLIST + +# ── Launcher executable (placeholders filled below) ── +cat > "$APP/Contents/MacOS/$APP_NAME.tmpl" <<'LAUNCHER' +#!/bin/bash +# Odysseus.app — start the local server and open the UI in an app window. +INSTALL_DIR="__INSTALL_DIR__" +PORT="__PORT__" +URL="http://127.0.0.1:${PORT}" +export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:$PATH" + +UVICORN="$INSTALL_DIR/venv/bin/uvicorn" +LOG="$INSTALL_DIR/logs/odysseus-app.log" + +notify() { /usr/bin/osascript -e "display notification \"$1\" with title \"Odysseus\"" >/dev/null 2>&1; } +die_gui() { + /usr/bin/osascript -e "display dialog \"$1\" with title \"Odysseus\" buttons {\"OK\"} default button 1 with icon stop" >/dev/null 2>&1 + exit 1 +} + +[ -x "$UVICORN" ] || die_gui "Odysseus isn't set up yet. Open Terminal and run: + +cd $INSTALL_DIR +python3.11 -m venv venv +./venv/bin/pip install -r requirements.txt +./venv/bin/python setup.py" + +# Open the UI in a chrome-less app window (Chromium browsers), else default browser. +open_ui() { + local b base exe bin + for b in "Google Chrome" "Microsoft Edge" "Brave Browser" "Chromium"; do + for base in "/Applications" "$HOME/Applications"; do + if [ -d "$base/$b.app" ]; then + exe="$(/usr/bin/defaults read "$base/$b.app/Contents/Info" CFBundleExecutable 2>/dev/null)" + bin="$base/$b.app/Contents/MacOS/$exe" + if [ -x "$bin" ]; then + "$bin" --app="$URL" --new-window >/dev/null 2>&1 & + return 0 + fi + fi + done + done + /usr/bin/open "$URL" +} + +mkdir -p "$INSTALL_DIR/logs" + +# Already running? Just open the UI. +if /usr/bin/curl -s -o /dev/null --max-time 2 "$URL"; then + open_ui + exit 0 +fi + +notify "Starting…" +cd "$INSTALL_DIR" || die_gui "Install folder not found: $INSTALL_DIR" +"$UVICORN" app:app --host 127.0.0.1 --port "$PORT" >>"$LOG" 2>&1 & +SERVER_PID=$! + +# Quitting the app stops the server it started. +trap 'kill $SERVER_PID 2>/dev/null; exit 0' TERM INT + +# Wait for readiness (first run downloads an embedding model — allow ~2 min). +READY=0 +for i in $(seq 1 120); do + /usr/bin/curl -s -o /dev/null --max-time 2 "$URL" && { READY=1; break; } + kill -0 "$SERVER_PID" 2>/dev/null || die_gui "Odysseus failed to start. Log: +$LOG" + sleep 1 +done + +if [ "$READY" = "1" ]; then + open_ui +else + notify "Odysseus is taking a while — open $URL once it finishes starting." +fi +wait "$SERVER_PID" +LAUNCHER + +sed -e "s|__INSTALL_DIR__|$INSTALL_DIR|g" -e "s|__PORT__|$PORT|g" \ + "$APP/Contents/MacOS/$APP_NAME.tmpl" > "$APP/Contents/MacOS/$APP_NAME" +rm -f "$APP/Contents/MacOS/$APP_NAME.tmpl" +chmod +x "$APP/Contents/MacOS/$APP_NAME" + +# Refresh Finder's icon cache for the new bundle. +touch "$APP" + +# ── .dmg (drag-to-Applications) ── +echo "Packaging dist/$APP_NAME.dmg" +STAGE="$(mktemp -d)/dmg" +mkdir -p "$STAGE" +cp -R "$APP" "$STAGE/" +ln -s /Applications "$STAGE/Applications" +rm -f "$DIST/$APP_NAME.dmg" +hdiutil create -volname "$APP_NAME" -srcfolder "$STAGE" -ov -format UDZO "$DIST/$APP_NAME.dmg" >/dev/null +rm -rf "$STAGE" + +echo "" +echo "Done:" +echo " $APP" +echo " $DIST/$APP_NAME.dmg" +echo "" +echo "Run it: open '$APP'" +echo "Install: open '$DIST/$APP_NAME.dmg' (drag Odysseus to Applications)" diff --git a/routes/cookbook_helpers.py b/routes/cookbook_helpers.py index 97ef2ca..a8412d5 100644 --- a/routes/cookbook_helpers.py +++ b/routes/cookbook_helpers.py @@ -102,6 +102,28 @@ def _shell_path(p: str) -> str: return '"' + p + '"' +def _local_tooling_path_export(executable: str) -> str: + """Bash line prepending the running interpreter's bin dir to PATH. + + When Odysseus runs from a virtualenv, that bin dir holds the tools the + cookbook runners shell out to (`hf`, `python`). tmux runners start from a + fresh login shell with the venv NOT activated, so without this they can't + find `hf` and downloads fail with "hf: command not found" — notably on + macOS, where the `pip --user` self-heal also misses (`pip` isn't a command, + only `pip3`/`python3 -m pip`). Local runs only; meaningless over SSH. + """ + bin_dir = os.path.dirname(os.path.abspath(executable)) + # Escape for a double-quoted context: $PATH must still expand, but spaces + # and shell metacharacters in the path must be preserved literally. + esc = ( + bin_dir.replace("\\", "\\\\") + .replace('"', '\\"') + .replace("$", "\\$") + .replace("`", "\\`") + ) + return f'export PATH="{esc}:$PATH"' + + def _ps_squote(v: str) -> str: """Escape a value for PowerShell single-quoted string interpolation. Belt-and-suspenders on top of _validate_token's regex — if the regex diff --git a/routes/cookbook_routes.py b/routes/cookbook_routes.py index 7a27146..921ed34 100644 --- a/routes/cookbook_routes.py +++ b/routes/cookbook_routes.py @@ -7,6 +7,7 @@ import os import re import shlex import shutil +import sys import uuid from pathlib import Path @@ -25,7 +26,7 @@ from routes.cookbook_helpers import ( _validate_repo_id, _validate_include, _validate_remote_host, _validate_token, _validate_local_dir, _validate_ssh_port, _validate_gpus, _shell_path, _ps_squote, _bash_squote, _validate_serve_cmd, _parse_serve_phase, - _safe_env_prefix, + _safe_env_prefix, _local_tooling_path_export, ModelDownloadRequest, ServeRequest, ) @@ -357,16 +358,22 @@ def setup_cookbook_routes() -> APIRouter: lines.append(f"export HF_TOKEN='{_bash_squote(req.hf_token)}'") # Ensure pip-user scripts (e.g. hf CLI installed via --user) are on PATH lines.append('export PATH="$HOME/.local/bin:$PATH"') + # When Odysseus runs from a venv (e.g. native macOS install), put its bin + # on PATH so the tmux shell finds the bundled `hf`/`python3` without an + # activated venv. Local bash runs only — meaningless over SSH/Windows. + if not req.remote_host and req.platform != "windows": + lines.append(_local_tooling_path_export(sys.executable)) # Best-effort install hf CLI (always). hf_transfer (Rust parallel downloader) # is fast but flaky on large files — it tends to crash near the end at high # throughput. Retries set disable_hf_transfer to fall back to the plain, # slower-but-reliable downloader (resumes cleanly from the .incomplete files). - lines.append("command -v hf >/dev/null 2>&1 || pip install --user --break-system-packages -q -U huggingface_hub 2>/dev/null || pip install -q -U huggingface_hub 2>/dev/null") + # Use `python3 -m pip` not `pip` — macOS has no bare `pip` command. + lines.append("command -v hf >/dev/null 2>&1 || python3 -m pip install --user --break-system-packages -q -U huggingface_hub 2>/dev/null || python3 -m pip install -q -U huggingface_hub 2>/dev/null") if req.disable_hf_transfer: lines.append("export HF_HUB_ENABLE_HF_TRANSFER=0") lines.append("export HF_HUB_DOWNLOAD_MAX_WORKERS=4") else: - lines.append("python3 -c 'import hf_transfer' 2>/dev/null || pip install --user --break-system-packages -q hf_transfer 2>/dev/null || pip install -q hf_transfer 2>/dev/null") + lines.append("python3 -c 'import hf_transfer' 2>/dev/null || python3 -m pip install --user --break-system-packages -q hf_transfer 2>/dev/null || python3 -m pip install -q hf_transfer 2>/dev/null") lines.append("python3 -c 'import hf_transfer' 2>/dev/null && export HF_HUB_ENABLE_HF_TRANSFER=1") lines.append("export HF_HUB_DOWNLOAD_MAX_WORKERS=8") @@ -845,6 +852,10 @@ def setup_cookbook_routes() -> APIRouter: # ── Linux/Termux: bash + tmux (existing flow) ── runner_lines = ["#!/bin/bash"] runner_lines.extend(_user_shell_path_bootstrap()) + # Put Odysseus's own venv bin on PATH (local runs only) so the serve + # shell resolves the bundled python3/hf, mirroring the download flow. + if not remote: + runner_lines.append(_local_tooling_path_export(sys.executable)) runner_lines.append("export FLASHINFER_DISABLE_VERSION_CHECK=1") if req.hf_token: runner_lines.append(f"export HF_TOKEN='{_bash_squote(req.hf_token)}'") @@ -864,7 +875,10 @@ def setup_cookbook_routes() -> APIRouter: # Jinja2 rejects (do_tojson ensure_ascii). Build it once from # source if missing; keep llama-cpp-python only as a fallback. runner_lines.append('# Ensure a llama.cpp server (prefer native llama-server)') - runner_lines.append('export PATH="$HOME/.local/bin:$HOME/bin:$HOME/llama.cpp/build/bin:$PATH"') + # Include the Homebrew bin dirs so a brew-installed llama-server / + # ollama is found (otherwise macOS falls back to a slow source build). + # /opt/homebrew = Apple Silicon, /usr/local = Intel; harmless on Linux. + runner_lines.append('export PATH="$HOME/.local/bin:$HOME/bin:$HOME/llama.cpp/build/bin:/opt/homebrew/bin:/usr/local/bin:$PATH"') runner_lines.append('if [ -d /data/data/com.termux ]; then') runner_lines.append(' # Termux: no native build — use the Python bindings (CPU).') runner_lines.append(' if ! python3 -c "import llama_cpp" 2>/dev/null; then') @@ -876,17 +890,50 @@ def setup_cookbook_routes() -> APIRouter: runner_lines.append(' echo "Native llama-server not found — building from source (one-time, may take a few minutes)..."') runner_lines.append(' mkdir -p ~/bin') runner_lines.append(' cd ~ && [ -d llama.cpp ] || git clone --depth 1 https://github.com/ggml-org/llama.cpp') - # GPU build if CUDA is present; fall back to a plain (CPU) build. - runner_lines.append(' cd ~/llama.cpp && { cmake -B build -DGGML_CUDA=ON 2>/dev/null || cmake -B build; } \\') - runner_lines.append(' && cmake --build build -j"$(nproc)" --target llama-server \\') - runner_lines.append(' && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server') + # Build with the right accelerator: Metal on macOS (llama.cpp + # enables it automatically, no flag), CUDA on Linux when present, + # else a plain CPU build. nproc is Linux-only — fall back to + # `sysctl hw.ncpu` on macOS. (Tip: `brew install llama.cpp` ships + # a prebuilt llama-server and skips this whole source build.) + runner_lines.append(' NPROC="$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)"') + runner_lines.append(' if [ "$(uname -s)" = "Darwin" ]; then') + runner_lines.append(' command -v cmake >/dev/null 2>&1 || echo "WARNING: cmake not found — install it with: brew install cmake (or: brew install llama.cpp for a prebuilt llama-server)."') + # Start from a clean cache: a prior failed configure (e.g. a CUDA + # attempt) poisons build/CMakeCache.txt, so a plain `cmake -B build` + # would reuse the bad settings and fail again. CMAKE_BUILD_TYPE is + # explicit so the binary is optimized (Metal auto-enables on macOS). + runner_lines.append(' cd ~/llama.cpp && rm -rf build && cmake -B build -DCMAKE_BUILD_TYPE=Release \\') + runner_lines.append(' && cmake --build build -j"$NPROC" --target llama-server \\') + runner_lines.append(' && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server') + runner_lines.append(' else') + runner_lines.append(' cd ~/llama.cpp && { cmake -B build -DGGML_CUDA=ON 2>/dev/null || cmake -B build; } \\') + runner_lines.append(' && cmake --build build -j"$NPROC" --target llama-server \\') + runner_lines.append(' && ln -sf ~/llama.cpp/build/bin/llama-server ~/bin/llama-server') + runner_lines.append(' fi') runner_lines.append(' # If the native build failed, fall back to the Python bindings.') runner_lines.append(' if ! command -v llama-server &>/dev/null && ! python3 -c "import llama_cpp" 2>/dev/null; then') runner_lines.append(' echo "llama-server build failed — installing Python bindings as fallback..."') runner_lines.append(' pip install --user --break-system-packages -q llama-cpp-python 2>/dev/null || pip install -q llama-cpp-python 2>/dev/null || true') runner_lines.append(' fi') runner_lines.append('fi') + elif "ollama" in req.cmd: + # Ollama manages its own model store and HTTP server. Just make + # sure the binary exists and the daemon is up before running the + # command (the natural serving engine on Apple Silicon / Metal). + runner_lines.append('if ! command -v ollama &>/dev/null; then') + runner_lines.append(' echo "ERROR: Ollama not found. Install it (macOS: brew install ollama, or https://ollama.com/download), then launch again."') + runner_lines.append(' exit 127') + runner_lines.append('fi') + runner_lines.append('if ! curl -sf http://localhost:11434/api/tags >/dev/null 2>&1; then') + runner_lines.append(' echo "Starting ollama server..."; (ollama serve >/dev/null 2>&1 &)') + runner_lines.append(' for _ in 1 2 3 4 5 6 7 8 9 10; do curl -sf http://localhost:11434/api/tags >/dev/null 2>&1 && break; sleep 1; done') + runner_lines.append('fi') elif "vllm serve" in req.cmd: + # vLLM is CUDA/ROCm-only and does not run on macOS at all. + runner_lines.append('if [ "$(uname -s)" = "Darwin" ]; then') + runner_lines.append(' echo "ERROR: vLLM does not run on macOS. Use Ollama or llama.cpp (Metal) instead."') + runner_lines.append(' exit 1') + runner_lines.append('fi') # Put ~/.local/bin on PATH first — without a venv, vllm installs # there via --user and the non-login serve shell otherwise can't # find the `vllm` CLI ("command not found"). Mirrors llama.cpp above. diff --git a/scripts/odysseus-cookbook b/scripts/odysseus-cookbook index 57edbce..845a2db 100755 --- a/scripts/odysseus-cookbook +++ b/scripts/odysseus-cookbook @@ -95,21 +95,89 @@ def cmd_list(args) -> None: # ─── gpus ──────────────────────────────────────────────────────────── +def _macos_metal_gpu() -> list | None: + """Apple Silicon has no discrete VRAM — report total unified memory as the + GPU budget so the web UI's picker shows the Mac's Metal GPU instead of + 'no GPU'. `free` is approximated from vm_stat (page-granular); macOS doesn't + expose Metal utilization to the shell, so util is 0. Returns None off macOS.""" + if sys.platform != "darwin": + return None + + def _sysctl(key: str) -> str | None: + try: + r = subprocess.run(["sysctl", "-n", key], capture_output=True, text=True, timeout=5) + return r.stdout.strip() if r.returncode == 0 else None + except Exception: + return None + + memsize = _sysctl("hw.memsize") + if not memsize or not memsize.isdigit(): + return None + total_mb = int(memsize) // (1024 * 1024) + name = _sysctl("machdep.cpu.brand_string") or "Apple Silicon" + + free_mb = total_mb + try: + vm = subprocess.run(["vm_stat"], capture_output=True, text=True, timeout=5) + if vm.returncode == 0: + page_size, pages = 4096, {} + for line in vm.stdout.splitlines(): + if "page size of" in line: + m = re.search(r"page size of (\d+)", line) + if m: + page_size = int(m.group(1)) + elif ":" in line: + k, v = line.split(":", 1) + v = v.strip().rstrip(".") + if v.isdigit(): + pages[k.strip()] = int(v) + free_pages = (pages.get("Pages free", 0) + pages.get("Pages inactive", 0) + + pages.get("Pages speculative", 0)) + if free_pages: + free_mb = (free_pages * page_size) // (1024 * 1024) + except Exception: + pass + + return [{ + "index": 0, + "name": name, + "free_mb": free_mb, + "total_mb": total_mb, + "used_mb": max(0, total_mb - free_mb), + "util_pct": 0, + "uuid": "apple-metal-0", + "unified_memory": True, + "busy": (free_mb / total_mb) < 0.5 if total_mb else False, + }] + + def cmd_gpus(args) -> None: """Same shape the web UI gets — index/name/free_mb/total_mb/used_mb/ - util_pct/uuid. Returns `[]` with an `error` field if nvidia-smi is - missing (laptop / CPU-only box). Pass `--host user@box` to run over - SSH against a remote machine.""" + util_pct/uuid. On Apple Silicon (no nvidia-smi) reports the Metal GPU's + unified memory instead. Returns `[]` with an `error` field only on a + CPU-only non-Mac box. Pass `--host user@box` to run over SSH.""" query = "nvidia-smi --query-gpu=index,name,memory.free,memory.total,memory.used,utilization.gpu,uuid --format=csv,noheader,nounits" prefix = _ssh_prefix(args.host, args.ssh_port) cmd = prefix + (query.split() if not prefix else [query]) try: out = subprocess.run(cmd, capture_output=True, text=True, timeout=15) except FileNotFoundError: + # No nvidia-smi locally → try the Metal fallback before giving up. + if not prefix: + mac = _macos_metal_gpu() + if mac is not None: + emit({"ok": True, "gpus": mac, "backend": "metal"}, args) + return msg = "ssh not found" if prefix else "nvidia-smi not found" emit({"ok": False, "error": msg, "gpus": []}, args) return if out.returncode != 0: + # nvidia-smi present but errored (or no NVIDIA GPU) — fall back to Metal. + if not prefix: + mac = _macos_metal_gpu() + if mac is not None: + emit({"ok": True, "gpus": mac, "backend": "metal"}, args) + return emit({"ok": False, "error": out.stderr.strip()[:200], "gpus": []}, args) return gpus = [] diff --git a/services/hwfit/fit.py b/services/hwfit/fit.py index 0cd142c..f3207f1 100644 --- a/services/hwfit/fit.py +++ b/services/hwfit/fit.py @@ -19,12 +19,22 @@ GPU_BANDWIDTH = { "6950 xt": 576, "6900 xt": 512, "6800 xt": 512, "6800": 512, "6700 xt": 384, "6600 xt": 256, "6600": 224, "mi300x": 5300, "mi300": 5300, "mi250x": 3277, "mi250": 3277, "mi210": 1638, "mi100": 1229, "9070 xt": 624, "9070": 488, + # Apple Silicon unified-memory bandwidth (GB/s). Keyed off the chip name + # reported by sysctl machdep.cpu.brand_string (e.g. "Apple M4 Max"). Listed + # before the bare "m_" keys matters less than length-sorting (done below), + # which guarantees "m4 max" is tried before "m4". + "m1 ultra": 800, "m1 max": 400, "m1 pro": 200, "m1": 68, + "m2 ultra": 800, "m2 max": 400, "m2 pro": 200, "m2": 100, + "m3 ultra": 800, "m3 max": 300, "m3 pro": 150, "m3": 100, + "m4 max": 410, "m4 pro": 273, "m4": 120, } # Pre-sort keys by length descending for correct substring matching _BW_KEYS_SORTED = sorted(GPU_BANDWIDTH.keys(), key=len, reverse=True) -FALLBACK_K = {"cuda": 220, "rocm": 180, "cpu_x86": 70, "cpu_arm": 90} +# metal: backstop for Apple Silicon chips not in GPU_BANDWIDTH (e.g. a future +# M5) — the named chips above take the accurate bandwidth path instead. +FALLBACK_K = {"cuda": 220, "rocm": 180, "metal": 150, "cpu_x86": 70, "cpu_arm": 90} USE_CASE_WEIGHTS = { "general": (0.45, 0.30, 0.15, 0.10), @@ -411,17 +421,28 @@ def rank_models(system, use_case=None, limit=50, search=None, sort="score", quan # If user picked a prequantized format (AWQ/FP8/GPTQ), filter to only those models filter_native = quant and any(quant.startswith(p) for p in ("AWQ-", "GPTQ-", "FP8")) - # MLX-quantized models only run on Apple Silicon (Metal). Exclude them on - # every other backend (CUDA / ROCm / CPU) so Linux/Windows users don't see - # unrunnable suggestions. system_backend = (system.get("backend") or "").lower() apple_silicon = system_backend in ("mps", "metal", "apple") for m in models: native_q = m.get("quantization", "") - # Drop MLX models on non-Apple hardware - if not apple_silicon and native_q.startswith("mlx-"): + # MLX-quantized models need the MLX runtime (mlx_lm), which Odysseus + # doesn't generate serve commands for — only llama.cpp/Ollama (Metal) + # and vLLM/SGLang (CUDA). MLX repos ship no GGUF alternative, so they're + # unrunnable on every backend we support. Always drop them, on Apple + # Silicon too, so the Cookbook never recommends a model it can't serve. + if native_q.startswith("mlx-"): + continue + + # On Apple Silicon the only serving engines are llama.cpp and Ollama, + # both GGUF-only (vLLM/SGLang are CUDA/ROCm and don't run on macOS). So + # a model is Metal-servable ONLY if it ships a real GGUF. Drop everything + # else — raw safetensors repos (which the catalog still tags with a + # default GGUF quant) and vLLM-only AWQ/GPTQ/FP8 builds alike. Without + # this the Cookbook recommends models the Mac can't run; on CUDA these + # stay visible because vLLM serves safetensors directly. + if apple_silicon and not (m.get("is_gguf") or m.get("gguf_sources")): continue # Format filter: AWQ tab → only AWQ models, FP8 tab → only FP8 models diff --git a/services/hwfit/hardware.py b/services/hwfit/hardware.py index 86aa777..c5ff486 100644 --- a/services/hwfit/hardware.py +++ b/services/hwfit/hardware.py @@ -204,6 +204,82 @@ def _detect_amd(): return None +def _detect_apple_silicon(): + """Detect Apple Silicon (M-series) GPUs. + + Macs have no discrete VRAM — the GPU shares the system's unified memory. + We report a fraction of total RAM as the usable GPU budget (matching macOS's + default Metal working-set limit) so the Cookbook recommends models that + actually run on the GPU instead of classifying the machine as CPU-only. + + backend="metal" is what services.hwfit.fit and the serve-command generation + key off of (they already understand MLX / llama.cpp-Metal). Works locally + (platform.system()=="Darwin") and over SSH (uname -s == Darwin). + """ + # Gate to macOS — locally via platform, remotely via uname. + if _remote_host: + if "darwin" not in (_run(["uname", "-s"]) or "").lower(): + return None + arch = (_run(["uname", "-m"]) or "").lower() + else: + if platform.system() != "Darwin": + return None + arch = platform.machine().lower() + + # Only Apple Silicon (arm64) has a Metal GPU worth serving LLMs on; Intel + # Macs fall through to the CPU path. + if "arm" not in arch and "aarch64" not in arch: + return None + + # Chip name, e.g. "Apple M4 Max" — carries the Pro/Max/Ultra variant that + # the fit bandwidth table keys off of. + brand = (_run(["sysctl", "-n", "machdep.cpu.brand_string"]) or "Apple Silicon").strip() + + # Total unified memory in bytes. + memsize = _run(["sysctl", "-n", "hw.memsize"]) + try: + total_gb = int(memsize) / (1024**3) if memsize else 0.0 + except ValueError: + total_gb = 0.0 + if total_gb <= 0: + return None + + # Usable GPU budget. macOS lets Metal use most of unified memory, but the + # default working-set limit scales with RAM: small machines have to keep + # more back for the OS + app. These fractions track Apple's + # recommendedMaxWorkingSetSize defaults across the lineup. Honour an + # explicit override if the user raised it with + # `sudo sysctl iogpu.wired_limit_mb=…`. + if total_gb <= 16: + frac = 0.67 + elif total_gb <= 64: + frac = 0.75 + else: + frac = 0.80 + vram_gb = round(total_gb * frac, 1) + wired = _run(["sysctl", "-n", "iogpu.wired_limit_mb"]) + try: + wired_mb = int(wired) if wired else 0 + if wired_mb > 0: + vram_gb = round(wired_mb / 1024.0, 1) + except ValueError: + pass + + gpu = {"index": 0, "name": brand, "vram_gb": vram_gb} + return { + "gpu_name": brand, + "gpu_vram_gb": vram_gb, + "gpu_count": 1, + "gpus": [gpu], + "gpu_groups": _group_gpus([gpu]), + "homogeneous": True, + "backend": "metal", + # Unified memory: the "VRAM" above is carved out of system RAM, not a + # separate pool — downstream fit logic uses this to avoid double-budgeting. + "unified_memory": True, + } + + def _read_file(path): """Read a file, locally or via SSH.""" if _remote_host: @@ -246,6 +322,15 @@ def _get_ram_gb(): return (pages * page_size) / (1024**3) except Exception: pass + + # macOS has no /proc/meminfo — fall back to sysctl (works locally and over + # SSH to a remote Mac, where the sysconf path above isn't taken). + memsize = _run(["sysctl", "-n", "hw.memsize"]) + if memsize: + try: + return int(memsize.strip()) / (1024**3) + except ValueError: + pass return 0.0 @@ -263,6 +348,12 @@ def _get_cpu_name(): if line.startswith("model name"): return line.split(":", 1)[1].strip() + # macOS has no /proc/cpuinfo — sysctl gives the chip name (e.g. "Apple M4"). + # Harmlessly returns nothing on Linux, so it's safe to try unconditionally. + brand = _run(["sysctl", "-n", "machdep.cpu.brand_string"]) + if brand and brand.strip(): + return brand.strip() + if not _remote_host: return platform.processor() or "unknown" return "unknown" @@ -270,7 +361,8 @@ def _get_cpu_name(): def _get_cpu_count(): if _remote_host: - out = _run(["nproc"]) + # nproc on Linux; hw.ncpu via sysctl on a remote Mac (no nproc there). + out = _run(["nproc"]) or _run(["sysctl", "-n", "hw.ncpu"]) if out: try: return int(out.strip()) @@ -411,7 +503,7 @@ def detect_system(host="", ssh_port="", platform="", fresh=False): cpu_cores = _get_cpu_count() cpu_name = _get_cpu_name() - gpu_info = _detect_nvidia() or _detect_amd() + gpu_info = _detect_apple_silicon() or _detect_nvidia() or _detect_amd() if gpu_info: result = { @@ -427,6 +519,9 @@ def detect_system(host="", ssh_port="", platform="", fresh=False): "gpu_groups": gpu_info.get("gpu_groups", []), "homogeneous": gpu_info.get("homogeneous", True), "backend": gpu_info["backend"], + # Apple Silicon / AMD APUs share system RAM with the GPU — carry the + # flag through so callers can tell unified from discrete VRAM. + "unified_memory": gpu_info.get("unified_memory", False), } else: if _remote_host: diff --git a/setup.py b/setup.py index e13e83f..fb5ba04 100644 --- a/setup.py +++ b/setup.py @@ -109,9 +109,12 @@ def check_deps(): print("\n [warn] tmux not found") print(" Cookbook uses tmux for background downloads and model serves.") print(" Install it with your OS package manager, for example:") - print(" sudo apt install tmux") - print(" sudo pacman -S tmux") - print(" sudo dnf install tmux") + if sys.platform == "darwin": + print(" brew install tmux") + else: + print(" sudo apt install tmux") + print(" sudo pacman -S tmux") + print(" sudo dnf install tmux") elif os.name != "nt": print(" [ok] tmux installed") @@ -142,9 +145,12 @@ def main(): print(f" [warn] Admin creation failed: {e}") print("\n=== Setup complete ===") - print(f"\nStart the server with:") - print(f" python -m uvicorn app:app --host 0.0.0.0 --port 7000") - print(f"\nThen open http://localhost:7000") + # start-macos.sh launches the server itself (on its own port) right after + # this, so suppress the manual hint there to avoid a contradictory URL. + if not os.getenv("ODYSSEUS_SKIP_RUN_HINT"): + print(f"\nStart the server with:") + print(f" python -m uvicorn app:app --host 127.0.0.1 --port 7000") + print(f"\nThen open http://localhost:7000") print(f"Login with the admin username and temporary password printed above.\n") diff --git a/start-macos.sh b/start-macos.sh new file mode 100755 index 0000000..595a4b5 --- /dev/null +++ b/start-macos.sh @@ -0,0 +1,139 @@ +#!/bin/bash +# Odysseus — one-command quick start for macOS (Apple Silicon). +# +# ./start-macos.sh +# +# Installs everything Odysseus needs via Homebrew, sets up a local Python +# environment, and launches the app — so a generic Mac user can run it without +# knowing anything about venvs, pip, or uvicorn. Safe to re-run; it skips work +# that's already done. +# +# Why native (not Docker): Cookbook serves models on whatever machine Odysseus +# runs on, and Docker on macOS is a Linux VM with no access to the Metal GPU. +# Running natively lets Cookbook detect and use your Mac's GPU. +set -e + +REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$REPO_DIR" + +PORT="${ODYSSEUS_PORT:-7860}" # 7860, not 7000 — macOS AirPlay Receiver holds 7000. + +# Friendly message on any failure — re-running is safe (every step is idempotent). +trap 'echo; echo "✗ Setup failed above. It is safe to re-run ./start-macos.sh."; exit 1' ERR + +echo "▶ Odysseus quick start for macOS" + +# Fail fast if the port is already taken (e.g. a previous run still running). +if (exec 3<>"/dev/tcp/127.0.0.1/$PORT") 2>/dev/null; then + echo "✗ Port $PORT is already in use. Stop what's using it, or pick another port:" + echo " ODYSSEUS_PORT=7900 ./start-macos.sh" + exit 1 +fi + +# 1. Homebrew — the macOS package manager. We can't safely auto-install it +# (it wants its own interactive confirmation), so point the user at it. +if ! command -v brew >/dev/null 2>&1; then + echo + echo "Homebrew is required but not installed. Install it (one command), then re-run this script:" + echo ' /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"' + echo + echo "More info: https://brew.sh" + exit 1 +fi + +# 2. Find a Python 3.11+ to build the environment with. +# On Apple Silicon we require an *arm64* interpreter (Homebrew's, under +# /opt/homebrew). A universal2 or x86 Python — e.g. the python.org installer +# at /usr/local — produces a venv whose compiled extensions get loaded as the +# wrong architecture when launched from the .app bundle (Cookbook then dies +# with "incompatible architecture"). So on arm64 we only look under +# /opt/homebrew and install Homebrew's python@3.11 if it's missing. On Intel +# (or non-mac) we just use whatever Python 3.11+ is on PATH. +PY="" +if [ "$(uname -m)" = "arm64" ]; then + cands="/opt/homebrew/bin/python3.13 /opt/homebrew/bin/python3.12 /opt/homebrew/bin/python3.11" +else + cands="python3 python3.13 python3.12 python3.11" +fi +for cand in $cands; do + p="$(command -v "$cand" 2>/dev/null)" || continue + if "$p" -c 'import sys; raise SystemExit(0 if sys.version_info[:2] >= (3, 11) else 1)' 2>/dev/null; then + PY="$p"; break + fi +done + +# System dependencies: +# - tmux : Cookbook runs model downloads/serves in the background +# - llama.cpp : a prebuilt, Metal-enabled llama-server so Cookbook can serve +# GGUF models on the GPU with no compile step +# - python@3.11 : installed only if no suitable (arm64) Python was found above +echo "▶ Installing dependencies (Homebrew)…" +if [ -n "$PY" ]; then + echo " (using $("$PY" --version 2>&1) at $PY)" + brew install tmux llama.cpp +else + brew install python@3.11 tmux llama.cpp + PY="$(command -v /opt/homebrew/bin/python3.11 || command -v python3.11 || true)" +fi + +if [ -z "$PY" ] || [ ! -x "$PY" ]; then + echo "✗ Couldn't find a Python 3.11+ to build the environment with." + echo " Check: ls /opt/homebrew/bin/python3* (or install one: brew install python@3.11)" + exit 1 +fi + +# 3. Python environment + dependencies (kept inside the repo, in venv/). +# Named `venv` to match the manual steps and build-macos-app.sh, so the +# clickable .app reuses this same environment. +if [ ! -d venv ]; then + echo "▶ Creating Python environment…" + "$PY" -m venv venv +fi +echo "▶ Installing Python packages (first run downloads a few — can take a few minutes)…" +./venv/bin/python -m pip install --quiet --upgrade pip +# Not --quiet: this is the slow step, so show progress (and any real errors). +./venv/bin/python -m pip install -r requirements.txt + +# 4. First-run setup: creates data dirs and prints an initial admin password +# the first time (idempotent — does nothing if already set up). Suppress its +# manual run hint — we launch the server ourselves just below. +echo "▶ Preparing Odysseus…" +ODYSSEUS_SKIP_RUN_HINT=1 ./venv/bin/python setup.py + +# 5. Launch. Bind to loopback only (safe default). +URL="http://127.0.0.1:$PORT" + +# Open the browser automatically once the server is accepting connections — so +# the URL isn't lost in the startup logs that keep scrolling. Runs in the +# background and is cleaned up when the server stops. Skip with +# ODYSSEUS_NO_OPEN=1 (e.g. over SSH / headless). +POLLER_PID="" +if [ -z "$ODYSSEUS_NO_OPEN" ] && command -v open >/dev/null 2>&1; then + ( + for _ in $(seq 1 90); do + if (exec 3<>"/dev/tcp/127.0.0.1/$PORT") 2>/dev/null; then + printf '\n' + printf ' ┌────────────────────────────────────────────┐\n' + printf ' │ ✓ Odysseus is ready — opening your browser │\n' + printf ' │ %-40s │\n' "$URL" + printf ' │ (Press Ctrl+C in this window to stop) │\n' + printf ' └────────────────────────────────────────────┘\n\n' + open "$URL" + break + fi + sleep 1 + done + ) & + POLLER_PID=$! +fi + +# Setup is done — drop the setup-failure handler, and clean up the background +# opener when the server exits or the user presses Ctrl+C. +trap - ERR +trap '[ -n "$POLLER_PID" ] && kill "$POLLER_PID" 2>/dev/null' EXIT INT TERM + +echo +echo "▶ Starting Odysseus — it will open in your browser at $URL" +echo " (this takes a few seconds; press Ctrl+C here to stop)" +echo +./venv/bin/python -m uvicorn app:app --host 127.0.0.1 --port "$PORT" diff --git a/static/js/cookbook.js b/static/js/cookbook.js index ce299c7..795bcf2 100644 --- a/static/js/cookbook.js +++ b/static/js/cookbook.js @@ -171,6 +171,13 @@ export function _isWindows(hostOrTask) { return _getPlatform(hostOrTask) === 'windows'; } +/** Check if the detected (local) hardware is Apple Silicon / Metal. Keys off the + * hardware probe's backend rather than a platform string, since a local Mac + * reports no platform but does report backend: "metal". */ +export function _isMetal() { + return ['metal', 'mps', 'apple'].includes(String(_hwfitCache?.system?.backend || '').toLowerCase()); +} + /** Detect model-specific vLLM optimizations */ function _detectModelOptimizations(modelName) { const n = (modelName || '').toLowerCase(); @@ -252,6 +259,13 @@ export function _detectBackend(model) { return { backend: 'llamacpp', label: 'llama.cpp' }; } + // Apple Silicon (Metal) → llama.cpp (GGUF). vLLM/SGLang are CUDA/ROCm-only and + // don't run on macOS; AWQ/GPTQ/FP8 (vLLM-only) models are already filtered out + // of metal Cookbook results, so llama.cpp is always the right engine here. + if (['metal', 'mps', 'apple'].includes(sysBackend)) { + return { backend: 'llamacpp', label: 'llama.cpp' }; + } + // AWQ / GPTQ / FP8 → vLLM if (/^AWQ|^GPTQ/.test(q) || q === 'FP8') { return { backend: 'vllm', label: 'vLLM' }; @@ -1764,6 +1778,7 @@ const shared = { _sshPrefix, _getPlatform, _isWindows, + _isMetal, _buildEnvPrefix, _buildServeCmd, _shellQuote, diff --git a/static/js/cookbookServe.js b/static/js/cookbookServe.js index e343fe6..8ee8c5c 100644 --- a/static/js/cookbookServe.js +++ b/static/js/cookbookServe.js @@ -16,6 +16,7 @@ let _getPort; let _sshPrefix; let _getPlatform; let _isWindows; +let _isMetal; let _buildEnvPrefix; let _buildServeCmd; let _shellQuote; @@ -382,6 +383,9 @@ function _rerenderCachedModels() { panelHtml += `
`; const _backendChoices = _isWindows() ? [['llamacpp','llama.cpp']] + : _isMetal() + // Diffusers (diffusion_server.py) is CUDA-only — omit it on Metal. + ? [['llamacpp','llama.cpp'],['ollama','Ollama']] : [['vllm','vLLM'],['sglang','SGLang'],['llamacpp','llama.cpp'],['diffusers','Diffusers']]; const backendOpts = _backendChoices.map(([v,l]) => ``).join(''); panelHtml += ``; @@ -1592,6 +1596,7 @@ export function initServe(shared) { _sshPrefix = shared._sshPrefix; _getPlatform = shared._getPlatform; _isWindows = shared._isWindows; + _isMetal = shared._isMetal; _buildEnvPrefix = shared._buildEnvPrefix; _buildServeCmd = shared._buildServeCmd; _shellQuote = shared._shellQuote; diff --git a/tests/test_cookbook_helpers.py b/tests/test_cookbook_helpers.py index 34119c7..9f15e59 100644 --- a/tests/test_cookbook_helpers.py +++ b/tests/test_cookbook_helpers.py @@ -1,7 +1,12 @@ import pytest from fastapi import HTTPException -from routes.cookbook_helpers import _safe_env_prefix, _validate_gpus, _validate_ssh_port +from routes.cookbook_helpers import ( + _local_tooling_path_export, + _safe_env_prefix, + _validate_gpus, + _validate_ssh_port, +) def test_safe_env_prefix_accepts_quoted_venv_path(): @@ -38,3 +43,18 @@ def test_validate_gpus_accepts_indexes_only(): assert _validate_gpus("0,1,2") == "0,1,2" with pytest.raises(HTTPException): _validate_gpus("0; rm -rf /") + + +def test_local_tooling_path_export_prepends_interpreter_bin(): + """The cookbook runners must see the venv's bin (where `hf`/`python` live) + so tmux shells can find them without an activated venv.""" + assert ( + _local_tooling_path_export("/opt/venv/bin/python") + == 'export PATH="/opt/venv/bin:$PATH"' + ) + + +def test_local_tooling_path_export_preserves_spaces_and_expands_path(): + line = _local_tooling_path_export("/Users/John Smith/.venv/bin/python3") + assert line == 'export PATH="/Users/John Smith/.venv/bin:$PATH"' + assert line.endswith(':$PATH"') # $PATH stays expandable in double quotes diff --git a/tests/test_hwfit_macos.py b/tests/test_hwfit_macos.py new file mode 100644 index 0000000..ca3b902 --- /dev/null +++ b/tests/test_hwfit_macos.py @@ -0,0 +1,129 @@ +"""macOS / Apple Silicon (Metal) support for Cookbook hardware-fit. + +Covers the Metal-specific behavior added for Apple Silicon and locks in the +guarantee that non-macOS (Linux/Windows) detection is unchanged. +""" + +from services.hwfit import hardware +from services.hwfit.fit import rank_models +from services.hwfit.models import get_models + + +def _metal_system(ram_gb=16.0, vram_gb=10.7): + return { + "has_gpu": True, + "backend": "metal", + "gpu_name": "Apple M2", + "gpu_vram_gb": vram_gb, + "gpu_count": 1, + "available_ram_gb": ram_gb * 0.7, + "total_ram_gb": ram_gb, + "unified_memory": True, + } + + +def _fake_sysctl(brand="Apple M2 Pro", memsize_gb=32, wired_mb=None): + def run(cmd): + joined = " ".join(cmd) + if "machdep.cpu.brand_string" in joined: + return brand + if "hw.memsize" in joined: + return str(int(memsize_gb * 1024**3)) + if "iogpu.wired_limit_mb" in joined: + return str(wired_mb) if wired_mb is not None else None + return None + return run + + +def test_mlx_models_hidden_on_metal(): + """MLX-quantized models can't be served by llama.cpp or Ollama (the only + Metal-capable engines Odysseus generates), so they must never be recommended + on Apple Silicon — even though the catalog tags them as Apple-only.""" + results = rank_models(_metal_system(), limit=900) + mlx = [m for m in results if str(m.get("quant", "")).startswith("mlx-")] + assert mlx == [], f"MLX models surfaced but cannot be served: {[m['name'] for m in mlx]}" + + +def _cuda_system(): + return { + "has_gpu": True, "backend": "cuda", "gpu_name": "NVIDIA RTX 4090", + "gpu_vram_gb": 24.0, "gpu_count": 1, "available_ram_gb": 32.0, "total_ram_gb": 64.0, + } + + +def test_mlx_hidden_on_cuda_backend_unchanged(): + """Regression guard: Linux/CUDA users never saw MLX before and still don't.""" + mlx = [m for m in rank_models(_cuda_system(), limit=900) if str(m.get("quant", "")).startswith("mlx-")] + assert mlx == [] + + +def test_only_gguf_models_recommended_on_metal(): + """llama.cpp and Ollama (the only Metal engines) need GGUF. Safetensors-only + repos — incl. vLLM-only AWQ/GPTQ/FP8 — can't be served on Metal, so every + model recommended on Apple Silicon must ship a servable GGUF.""" + catalog = {m["name"]: m for m in get_models()} + unservable = [ + r["name"] for r in rank_models(_metal_system(), limit=900) + if not (catalog.get(r["name"], {}).get("is_gguf") + or catalog.get(r["name"], {}).get("gguf_sources")) + ] + assert unservable == [], f"{len(unservable)} non-GGUF models on Metal, e.g. {unservable[:3]}" + + +def test_safetensors_models_still_recommended_on_cuda(): + """Regression guard: vLLM serves safetensors on CUDA, so non-GGUF repos must + NOT be filtered there — the GGUF-only rule is Metal-specific.""" + names = {r["name"] for r in rank_models(_cuda_system(), limit=900)} + assert "microsoft/Phi-mini-MoE-instruct" in names + + +def test_apple_silicon_detected_as_metal(monkeypatch): + """On local Apple Silicon, detection reports a Metal GPU with a RAM-scaled + unified-memory budget.""" + monkeypatch.setattr(hardware, "_remote_host", None) + monkeypatch.setattr(hardware.platform, "system", lambda: "Darwin") + monkeypatch.setattr(hardware.platform, "machine", lambda: "arm64") + monkeypatch.setattr(hardware, "_run", _fake_sysctl(memsize_gb=32)) + + info = hardware._detect_apple_silicon() + assert info is not None + assert info["backend"] == "metal" + assert info["gpu_name"] == "Apple M2 Pro" + assert info["unified_memory"] is True + assert info["gpu_vram_gb"] == 24.0 # 32GB * 0.75 + + +def test_apple_silicon_skipped_on_linux(monkeypatch): + """Guarantee Linux detection is untouched: the Metal probe bails immediately.""" + monkeypatch.setattr(hardware, "_remote_host", None) + monkeypatch.setattr(hardware.platform, "system", lambda: "Linux") + monkeypatch.setattr(hardware.platform, "machine", lambda: "x86_64") + monkeypatch.setattr(hardware, "_run", _fake_sysctl()) + assert hardware._detect_apple_silicon() is None + + +def test_intel_mac_skipped(monkeypatch): + """Intel Macs have no Metal GPU worth serving LLMs on — fall through to CPU.""" + monkeypatch.setattr(hardware, "_remote_host", None) + monkeypatch.setattr(hardware.platform, "system", lambda: "Darwin") + monkeypatch.setattr(hardware.platform, "machine", lambda: "x86_64") + monkeypatch.setattr(hardware, "_run", _fake_sysctl()) + assert hardware._detect_apple_silicon() is None + + +def test_detect_system_propagates_unified_memory(monkeypatch): + """The unified_memory flag set by GPU detection must survive into the + system dict so the API and UI can report it (it was being dropped).""" + monkeypatch.setattr(hardware, "_detect_apple_silicon", lambda: { + "gpu_name": "Apple M4", "gpu_vram_gb": 10.7, "gpu_count": 1, + "gpus": [], "gpu_groups": [], "homogeneous": True, + "backend": "metal", "unified_memory": True, + }) + monkeypatch.setattr(hardware, "_get_ram_gb", lambda: 16.0) + monkeypatch.setattr(hardware, "_get_available_ram_gb", lambda: 11.0) + monkeypatch.setattr(hardware, "_get_cpu_count", lambda: 10) + monkeypatch.setattr(hardware, "_get_cpu_name", lambda: "Apple M4") + + s = hardware.detect_system(fresh=True) + assert s["backend"] == "metal" + assert s.get("unified_memory") is True