From 9112861d8ed8eafd964ba320f3fa612187d27567 Mon Sep 17 00:00:00 2001 From: pewdiepie-archdaemon Date: Thu, 4 Jun 2026 23:27:18 +0900 Subject: [PATCH] cookbook agent debug loop: persistent log files, auto-adopt orphan tmux, Codex/Claude skill parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three converging fixes so the chat agent + external Codex/Claude skills can actually debug a crashed serve instead of staring at a post-crash neofetch banner: * Serves now `tee` to /tmp/odysseus-tmux/SESSION.log on the host running them. Runner saves fds 3/4 before the tee and restores them right before `exec ${SHELL}`, so the post-crash interactive zsh banner does NOT pollute the log file. * `tail_serve_output` (chat agent) and `/api/codex/cookbook/output/{sid}` (Codex+Claude skills) both prefer the persistent log file over the tmux pane. Pane is fallback for sessions predating the tee runner. Default tail bumped 150 -> 400. * `list_served_models` "recent log" snippet seeks to the Traceback line instead of showing the last 6 lines (which was always the bash prompt). Cookbook auto-adoption sweep on `/api/cookbook/tasks/status`: every 20s (rate-limited) the cookbook SSHes each configured server, finds `serve-*` / `cookbook-*` tmux sessions running an actual model process (vllm/python/llama-server/etc., filtered via `pane_current_command`), and writes them into state.tasks. So when the agent falls back to raw ssh+tmux, the session appears in the Cookbook UI on the next poll. `serve_model` error path now reads `data["detail"]` in addition to `data["error"]` so the FastAPI HTTPException message ("Invalid characters in cmd") actually reaches the agent instead of being swallowed as a generic "Serve failed". Tool description updated to warn against `cd …`/`source …`/`&&` prefixes. Intent-without-action supervisor in agent_loop: when the model writes "Let me tail the output" / "I'll check the logs" / "Let me investigate" and ends the turn without emitting a tool call, the loop injects a sharp system nudge ("You said you would X — DO IT NOW") and continues. Capped at 2 nudges per chat so a model that genuinely cannot use the tool does not pin the loop. Codex/Claude skill parity: adds `/cookbook/cached`, `/cookbook/presets`, `/cookbook/preset/{name}`, `/cookbook/adopt` so external agents have the same surface as the chat agent. SKILL.md docs + odysseus_api.py wrapper updated for both bundles. `adopt_served_model` promoted to the always-on tool set so the agent has a documented fallback when serve_model rejects a cmd. Also various cookbook UI tweaks accumulated alongside the above (cookbook.js, cookbookRunning.js, cookbookServe.js, cookbook-diagnosis.js, settings.js, style.css). --- integrations/claude/skills/odysseus/SKILL.md | 45 ++- .../skills/odysseus/scripts/odysseus_api.py | 64 +++ integrations/codex/scripts/odysseus_api.py | 64 +++ integrations/codex/skills/odysseus/SKILL.md | 33 +- routes/codex_routes.py | 375 ++++++++++++++++++ routes/cookbook_helpers.py | 13 +- routes/cookbook_routes.py | 185 ++++++++- src/agent_loop.py | 63 +++ src/agent_tools.py | 2 +- src/tool_execution.py | 4 + src/tool_implementations.py | 367 +++++++++++++++-- src/tool_index.py | 13 +- src/tool_schemas.py | 15 + static/js/cookbook-diagnosis.js | 159 ++++---- static/js/cookbook.js | 83 +++- static/js/cookbookRunning.js | 132 +++++- static/js/cookbookServe.js | 55 ++- static/js/settings.js | 3 + static/style.css | 5 + 19 files changed, 1529 insertions(+), 151 deletions(-) diff --git a/integrations/claude/skills/odysseus/SKILL.md b/integrations/claude/skills/odysseus/SKILL.md index 3877f9c..d3b55b3 100644 --- a/integrations/claude/skills/odysseus/SKILL.md +++ b/integrations/claude/skills/odysseus/SKILL.md @@ -1,6 +1,6 @@ --- name: odysseus -description: Use when the user asks Claude Code to read or write Odysseus data (todos, email, calendar, memory, documents) through the scoped Claude Agent API. Requires ODYSSEUS_URL and ODYSSEUS_API_TOKEN. +description: Use when the user asks Claude Code to read or write Odysseus data (todos, email, calendar, memory, documents) or to launch/monitor/stop a Cookbook model-serve task through the scoped Claude Agent API. Requires ODYSSEUS_URL and ODYSSEUS_API_TOKEN. --- # Odysseus @@ -105,6 +105,49 @@ python3 ~/.claude/skills/odysseus/scripts/odysseus_api.py POST /api/codex/memory - `POST /api/codex/emails/draft` — body matches `SendEmailRequest` (`to`, `cc`, `bcc`, `subject`, `body`, `body_html`, `attachments`, `account_id`, `in_reply_to`, `references`). Requires `email:draft` (or `email:send`). - `POST /api/codex/emails/send` — same body. Requires `email:send`. Never send without explicit user instruction. +## Cookbook serve (debug a failing model launch) + +The Cookbook surface lets you reproduce what a human would do in Odysseus → Cookbook: read which serves are running, tail their tmux output to see why they crashed, edit the launch command, relaunch, kill a stuck one. Use this when the user is debugging a model server that won't come up (compute-capability errors, OOM, missing kernels, wrong attention backend, etc.). + +- `GET /api/codex/cookbook/tasks` — list active serve/download/install tasks (sessionId, type, status, repo_id, remoteHost, payload._cmd). Requires `cookbook:read`. +- `GET /api/codex/cookbook/servers` — list configured servers (name, host, port, env type + path, model dirs). Requires `cookbook:read`. +- `GET /api/codex/cookbook/cached?host=` — list models already cached on the named server (HF cache + Ollama + extra modelDirs). Call BEFORE `serve` to see what's already on disk. Requires `cookbook:read`. +- `GET /api/codex/cookbook/presets` — list saved serve presets (model + host + port + cmd). The user's saved preset usually has a working cmd — try `preset NAME` before composing your own. Requires `cookbook:read`. +- `GET /api/codex/cookbook/output/{session_id}?tail=400` — read the last N lines of the task's persistent log file (preferred) or tmux pane (fallback). The log file persists across vllm crashes, so this returns the actual Python traceback even after the bash prompt + neofetch banner overwrites the pane. Default tail=400. Requires `cookbook:read`. +- `POST /api/codex/cookbook/serve` — launch a serve task. Body matches `ServeRequest`: `{ repo_id, cmd, remote_host?, ssh_port?, env_prefix?, gpus?, platform? }`. The `cmd` is validated: leading binary must be `vllm`/`python3`/`sglang`/`llama-server`/`ollama`/`node`/`npx`. NEVER prefix with `cd …`, `source …`, or chain with `&&`/`||`/`;`/`$(...)` — the validator rejects shell metacharacters. The venv activation (`env_prefix`) is added automatically from the host's saved settings, so pass the bare binary + args. Requires `cookbook:launch`. +- `POST /api/codex/cookbook/preset/{name}` — launch a saved preset by name. Reuses the working cmd + host the user already saved. Requires `cookbook:launch`. +- `POST /api/codex/cookbook/adopt` — register an externally-launched tmux session into cookbook tracking. Body: `{ tmux_session, model, host?, port? }`. Use this when serve_model rejected a cmd and you fell back to direct ssh+tmux — without adoption, the session is invisible to the UI. Requires `cookbook:launch`. +- `POST /api/codex/cookbook/stop/{session_id}` — kill the tmux session for that task. Requires `cookbook:launch`. + +```bash +# Survey what's running +python3 ~/.claude/skills/odysseus/scripts/odysseus_api.py cookbook tasks + +# Tail the failing one (sessionId from `cookbook tasks`) +python3 ~/.claude/skills/odysseus/scripts/odysseus_api.py cookbook output serve-abc12345 400 + +# Stop the previous attempt before you try a new flag set +python3 ~/.claude/skills/odysseus/scripts/odysseus_api.py cookbook stop serve-abc12345 + +# Relaunch with new flags. cmd MUST begin with one of the allowlisted binaries. +python3 ~/.claude/skills/odysseus/scripts/odysseus_api.py cookbook serve \ + /mnt/HADES/models/Qwen3.5-397B-A17B-AWQ \ + "vllm serve /mnt/HADES/models/Qwen3.5-397B-A17B-AWQ --host 0.0.0.0 --port 8001 --tensor-parallel-size 8 --max-model-len 262144 --gpu-memory-utilization 0.90 --dtype auto --max-num-seqs 8 --trust-remote-code --enable-expert-parallel --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3" \ + pewds@192.168.1.12 +``` + +**Debug loop pattern:** when a serve is failing, the productive sequence is + +1. `cookbook tasks` → find the failing sessionId. +2. `cookbook output SID 600` → read the last 600 lines, find the actual root-cause line (often above the visible tail because tmux scrollback rolled — request a larger `tail` if the error references "above"). +3. `cookbook stop SID` — kill the previous attempt before relaunching; two serves on the same `--port` collide. +4. `cookbook serve repo "new cmd"` — try the next variation. Wait ~20s, then `cookbook output` on the new sessionId. + +**Hard limits this surface enforces:** +- `cookbook serve` cmd allowlist + shell-metacharacter rejection — you cannot run arbitrary shell, only model-server binaries. +- `cookbook stop` only targets task sessionIds matching `[a-zA-Z0-9_-]+`. +- The agent CAN spawn GPU-pinning long-lived processes — always `cookbook stop` your previous attempt before relaunching, and check `cookbook tasks` for collisions on the same `--port` before launching. + ## Forbidden Bypass Pattern If you are about to reach the Odysseus host/container, import app internals, query the database, or call MCP helper modules directly, stop. Those paths bypass Odysseus Settings and token scopes. Ask the user to enable the relevant Claude Agent tool toggle instead. diff --git a/integrations/claude/skills/odysseus/scripts/odysseus_api.py b/integrations/claude/skills/odysseus/scripts/odysseus_api.py index 5fdd632..fcef8a7 100755 --- a/integrations/claude/skills/odysseus/scripts/odysseus_api.py +++ b/integrations/claude/skills/odysseus/scripts/odysseus_api.py @@ -17,6 +17,15 @@ def _usage() -> int: print(" odysseus_api.py todos add TITLE", file=sys.stderr) print(" odysseus_api.py emails list [limit]", file=sys.stderr) print(" odysseus_api.py emails read UID", file=sys.stderr) + print(" odysseus_api.py cookbook tasks", file=sys.stderr) + print(" odysseus_api.py cookbook servers", file=sys.stderr) + print(" odysseus_api.py cookbook cached [HOST]", file=sys.stderr) + print(" odysseus_api.py cookbook presets", file=sys.stderr) + print(" odysseus_api.py cookbook output SESSION_ID [tail]", file=sys.stderr) + print(" odysseus_api.py cookbook serve REPO_ID 'CMD' [REMOTE_HOST]", file=sys.stderr) + print(" odysseus_api.py cookbook preset NAME", file=sys.stderr) + print(" odysseus_api.py cookbook adopt SESSION_ID MODEL [HOST] [PORT]", file=sys.stderr) + print(" odysseus_api.py cookbook stop SESSION_ID", file=sys.stderr) print(" odysseus_api.py METHOD /api/codex/path [json-body]", file=sys.stderr) return 2 @@ -72,6 +81,61 @@ def main() -> int: body = None else: return _usage() + elif command == "cookbook": + if len(sys.argv) < 3: + return _usage() + action = sys.argv[2].lower() + if action == "tasks": + method = "GET" + path = "/api/codex/cookbook/tasks" + body = None + elif action == "servers": + method = "GET" + path = "/api/codex/cookbook/servers" + body = None + elif action == "output" and len(sys.argv) >= 4: + method = "GET" + sid = sys.argv[3] + tail = sys.argv[4] if len(sys.argv) >= 5 else "400" + path = f"/api/codex/cookbook/output/{sid}?tail={tail}" + body = None + elif action == "cached": + method = "GET" + if len(sys.argv) >= 4: + from urllib.parse import quote + path = f"/api/codex/cookbook/cached?host={quote(sys.argv[3])}" + else: + path = "/api/codex/cookbook/cached" + body = None + elif action == "presets": + method = "GET" + path = "/api/codex/cookbook/presets" + body = None + elif action == "preset" and len(sys.argv) >= 4: + from urllib.parse import quote + method = "POST" + path = f"/api/codex/cookbook/preset/{quote(sys.argv[3])}" + body = None + elif action == "adopt" and len(sys.argv) >= 5: + method = "POST" + path = "/api/codex/cookbook/adopt" + payload = {"tmux_session": sys.argv[3], "model": sys.argv[4]} + if len(sys.argv) >= 6: payload["host"] = sys.argv[5] + if len(sys.argv) >= 7: payload["port"] = int(sys.argv[6]) + body = json.dumps(payload) + elif action == "serve" and len(sys.argv) >= 5: + method = "POST" + path = "/api/codex/cookbook/serve" + payload = {"repo_id": sys.argv[3], "cmd": sys.argv[4]} + if len(sys.argv) >= 6: + payload["remote_host"] = sys.argv[5] + body = json.dumps(payload) + elif action == "stop" and len(sys.argv) >= 4: + method = "POST" + path = f"/api/codex/cookbook/stop/{sys.argv[3]}" + body = None + else: + return _usage() else: if len(sys.argv) < 3: return _usage() diff --git a/integrations/codex/scripts/odysseus_api.py b/integrations/codex/scripts/odysseus_api.py index 5fdd632..fcef8a7 100755 --- a/integrations/codex/scripts/odysseus_api.py +++ b/integrations/codex/scripts/odysseus_api.py @@ -17,6 +17,15 @@ def _usage() -> int: print(" odysseus_api.py todos add TITLE", file=sys.stderr) print(" odysseus_api.py emails list [limit]", file=sys.stderr) print(" odysseus_api.py emails read UID", file=sys.stderr) + print(" odysseus_api.py cookbook tasks", file=sys.stderr) + print(" odysseus_api.py cookbook servers", file=sys.stderr) + print(" odysseus_api.py cookbook cached [HOST]", file=sys.stderr) + print(" odysseus_api.py cookbook presets", file=sys.stderr) + print(" odysseus_api.py cookbook output SESSION_ID [tail]", file=sys.stderr) + print(" odysseus_api.py cookbook serve REPO_ID 'CMD' [REMOTE_HOST]", file=sys.stderr) + print(" odysseus_api.py cookbook preset NAME", file=sys.stderr) + print(" odysseus_api.py cookbook adopt SESSION_ID MODEL [HOST] [PORT]", file=sys.stderr) + print(" odysseus_api.py cookbook stop SESSION_ID", file=sys.stderr) print(" odysseus_api.py METHOD /api/codex/path [json-body]", file=sys.stderr) return 2 @@ -72,6 +81,61 @@ def main() -> int: body = None else: return _usage() + elif command == "cookbook": + if len(sys.argv) < 3: + return _usage() + action = sys.argv[2].lower() + if action == "tasks": + method = "GET" + path = "/api/codex/cookbook/tasks" + body = None + elif action == "servers": + method = "GET" + path = "/api/codex/cookbook/servers" + body = None + elif action == "output" and len(sys.argv) >= 4: + method = "GET" + sid = sys.argv[3] + tail = sys.argv[4] if len(sys.argv) >= 5 else "400" + path = f"/api/codex/cookbook/output/{sid}?tail={tail}" + body = None + elif action == "cached": + method = "GET" + if len(sys.argv) >= 4: + from urllib.parse import quote + path = f"/api/codex/cookbook/cached?host={quote(sys.argv[3])}" + else: + path = "/api/codex/cookbook/cached" + body = None + elif action == "presets": + method = "GET" + path = "/api/codex/cookbook/presets" + body = None + elif action == "preset" and len(sys.argv) >= 4: + from urllib.parse import quote + method = "POST" + path = f"/api/codex/cookbook/preset/{quote(sys.argv[3])}" + body = None + elif action == "adopt" and len(sys.argv) >= 5: + method = "POST" + path = "/api/codex/cookbook/adopt" + payload = {"tmux_session": sys.argv[3], "model": sys.argv[4]} + if len(sys.argv) >= 6: payload["host"] = sys.argv[5] + if len(sys.argv) >= 7: payload["port"] = int(sys.argv[6]) + body = json.dumps(payload) + elif action == "serve" and len(sys.argv) >= 5: + method = "POST" + path = "/api/codex/cookbook/serve" + payload = {"repo_id": sys.argv[3], "cmd": sys.argv[4]} + if len(sys.argv) >= 6: + payload["remote_host"] = sys.argv[5] + body = json.dumps(payload) + elif action == "stop" and len(sys.argv) >= 4: + method = "POST" + path = f"/api/codex/cookbook/stop/{sys.argv[3]}" + body = None + else: + return _usage() else: if len(sys.argv) < 3: return _usage() diff --git a/integrations/codex/skills/odysseus/SKILL.md b/integrations/codex/skills/odysseus/SKILL.md index 1e2be01..4cff140 100644 --- a/integrations/codex/skills/odysseus/SKILL.md +++ b/integrations/codex/skills/odysseus/SKILL.md @@ -1,6 +1,6 @@ --- name: odysseus -description: Use when the user asks Codex to read or write Odysseus data from a terminal Codex session through the scoped Codex Agent API. Requires ODYSSEUS_URL and ODYSSEUS_API_TOKEN. +description: Use when the user asks Codex to read or write Odysseus data (todos, email, calendar, memory, documents) or to launch/monitor/stop a Cookbook model-serve task through the scoped Codex Agent API. Requires ODYSSEUS_URL and ODYSSEUS_API_TOKEN. --- # Odysseus @@ -105,6 +105,37 @@ python3 integrations/codex/scripts/odysseus_api.py POST /api/codex/memory '{"tex - `POST /api/codex/emails/draft` — body matches `SendEmailRequest` (`to`, `cc`, `bcc`, `subject`, `body`, `body_html`, `attachments`, `account_id`, `in_reply_to`, `references`). Requires `email:draft` (or `email:send`). - `POST /api/codex/emails/send` — same body. Requires `email:send`. Never send without explicit user instruction. +## Cookbook serve (debug a failing model launch) + +The Cookbook surface lets you reproduce what a human would do in Odysseus → Cookbook: read which serves are running, tail their tmux output to see why they crashed, edit the launch command, relaunch, kill a stuck one. Use this when the user is debugging a model server that won't come up (compute-capability errors, OOM, missing kernels, wrong attention backend, etc.). + +- `GET /api/codex/cookbook/tasks` — list active serve/download/install tasks (sessionId, type, status, repo_id, remoteHost, payload._cmd). Requires `cookbook:read`. +- `GET /api/codex/cookbook/servers` — list configured servers (name, host, port, env type + path, model dirs). Requires `cookbook:read`. +- `GET /api/codex/cookbook/cached?host=` — list models already cached on the named server (HF cache + Ollama + extra modelDirs). Call BEFORE `serve` to see what's already on disk. Requires `cookbook:read`. +- `GET /api/codex/cookbook/presets` — list saved serve presets (model + host + port + cmd). The user's saved preset usually has a working cmd — try `preset NAME` before composing your own. Requires `cookbook:read`. +- `GET /api/codex/cookbook/output/{session_id}?tail=400` — read the last N lines of the task's persistent log file (preferred) or tmux pane (fallback). The log file persists across vllm crashes, so this returns the actual Python traceback even after the bash prompt + neofetch banner overwrites the pane. Default tail=400. Requires `cookbook:read`. +- `POST /api/codex/cookbook/serve` — launch a serve task. Body matches `ServeRequest`: `{ repo_id, cmd, remote_host?, ssh_port?, env_prefix?, gpus?, platform? }`. The `cmd` is validated: leading binary must be `vllm`/`python3`/`sglang`/`llama-server`/`ollama`/`node`/`npx`. NEVER prefix with `cd …`, `source …`, or chain with `&&`/`||`/`;`/`$(...)` — the validator rejects shell metacharacters. The venv activation (`env_prefix`) is added automatically from the host's saved settings, so pass the bare binary + args. Requires `cookbook:launch`. +- `POST /api/codex/cookbook/preset/{name}` — launch a saved preset by name. Reuses the working cmd + host the user already saved. Requires `cookbook:launch`. +- `POST /api/codex/cookbook/adopt` — register an externally-launched tmux session into cookbook tracking. Body: `{ tmux_session, model, host?, port? }`. Use this when serve_model rejected a cmd and you fell back to direct ssh+tmux — without adoption, the session is invisible to the UI. Requires `cookbook:launch`. +- `POST /api/codex/cookbook/stop/{session_id}` — kill the tmux session. Requires `cookbook:launch`. + +```bash +python3 ~/plugins/odysseus/scripts/odysseus_api.py cookbook tasks +python3 ~/plugins/odysseus/scripts/odysseus_api.py cookbook output serve-abc12345 400 +python3 ~/plugins/odysseus/scripts/odysseus_api.py cookbook stop serve-abc12345 +python3 ~/plugins/odysseus/scripts/odysseus_api.py cookbook serve \ + /mnt/HADES/models/Qwen3.5-397B-A17B-AWQ \ + "vllm serve /mnt/HADES/models/Qwen3.5-397B-A17B-AWQ --host 0.0.0.0 --port 8001 --tensor-parallel-size 8 --max-model-len 262144 --gpu-memory-utilization 0.90 --dtype auto --max-num-seqs 8 --trust-remote-code --enable-expert-parallel --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3" \ + pewds@192.168.1.12 +``` + +**Debug loop pattern:** `tasks` → `output SID 600` (find root cause; request larger `tail` if it references "above") → `stop SID` → `serve repo "new cmd"` → wait ~20s → `output` on the new sessionId. + +**Hard limits this surface enforces:** +- `cookbook serve` cmd allowlist + shell-metacharacter rejection. +- `cookbook stop` requires sessionIds matching `[a-zA-Z0-9_-]+`. +- Agent CAN spawn GPU-pinning long-lived processes — always `cookbook stop` your previous attempt before relaunching. + ## Forbidden Bypass Pattern If you are about to reach the Odysseus host/container, import app internals, query the database, or call MCP helper modules directly, stop. Those paths bypass Odysseus Settings and token scopes. Ask the user to enable the relevant Codex Agent tool toggle instead. diff --git a/routes/codex_routes.py b/routes/codex_routes.py index 8c59ee5..9898dae 100644 --- a/routes/codex_routes.py +++ b/routes/codex_routes.py @@ -19,6 +19,8 @@ from src.auth_helpers import require_user from src.tool_implementations import do_manage_notes +COOKBOOK_READ_SCOPES = {"cookbook:read", "cookbook:launch"} +COOKBOOK_LAUNCH_SCOPES = {"cookbook:launch"} TODO_READ_SCOPES = {"todos:read", "todos:write"} TODO_WRITE_SCOPES = {"todos:write"} EMAIL_READ_SCOPES = {"email:read", "email:draft", "email:send"} @@ -130,6 +132,11 @@ def setup_codex_routes( "actions": ["library", "read", "create", "delete"], "available": documents_library_endpoint is not None, }, + "cookbook": { + "read": scoped(COOKBOOK_READ_SCOPES), + "launch": scoped(COOKBOOK_LAUNCH_SCOPES), + "actions": ["tasks", "servers", "output", "serve", "stop"], + }, }, "safety": { "email_send_requires_confirmation": True, @@ -373,6 +380,374 @@ def setup_codex_routes( raise HTTPException(400, f"Invalid document payload: {exc}") return await _as_owner(request, owner, documents_create_endpoint, request, req) + # ── Cookbook surface ── + # Lets the agent run the same launch / monitor / kill loop the user + # would do by hand in the Cookbook UI: read the current task list + + # tmux output, launch a serve task, stop one. Two scopes: + # cookbook:read — list tasks + tail output + list servers + # cookbook:launch — also start/stop serves (host shell exec) + # `cookbook:launch` is genuinely powerful: /api/model/serve runs SSH'd + # commands on the user's hosts. The existing _validate_serve_cmd + # allowlist (vllm/python3/sglang/llama-server/etc., no shell metachars) + # keeps the agent inside the same sandbox the UI uses. + + async def _run_shell(cmd: str, timeout: float = 15.0) -> dict: + """Run a shell command, return {exit_code, stdout, stderr}.""" + import asyncio as _asyncio + try: + proc = await _asyncio.create_subprocess_shell( + cmd, + stdout=_asyncio.subprocess.PIPE, + stderr=_asyncio.subprocess.PIPE, + ) + try: + stdout_b, stderr_b = await _asyncio.wait_for(proc.communicate(), timeout=timeout) + except _asyncio.TimeoutError: + proc.kill() + return {"exit_code": -1, "stdout": "", "stderr": "timed out"} + return { + "exit_code": proc.returncode, + "stdout": stdout_b.decode(errors="replace"), + "stderr": stderr_b.decode(errors="replace"), + } + except Exception as exc: + return {"exit_code": -1, "stdout": "", "stderr": str(exc)} + + def _read_cookbook_state() -> dict: + from pathlib import Path as _Path + import os as _os, json as _json + p = _Path(_os.environ.get("DATA_DIR", "data")) / "cookbook_state.json" + if not p.exists(): + return {} + try: + return _json.loads(p.read_text(encoding="utf-8")) + except Exception: + return {} + + def _redact_task(t: dict) -> dict: + """Strip secrets before returning to the agent.""" + clean = {k: v for k, v in t.items() if k not in ("hf_token", "_secrets")} + if isinstance(clean.get("payload"), dict): + pl = clean["payload"] + clean["payload"] = {k: v for k, v in pl.items() + if k not in ("hf_token", "_secrets")} + return clean + + @router.get("/cookbook/tasks") + async def codex_cookbook_tasks(request: Request): + _scope_owner(request, COOKBOOK_READ_SCOPES) + state = _read_cookbook_state() + tasks = state.get("tasks") or [] + return {"tasks": [_redact_task(t) for t in tasks]} + + @router.get("/cookbook/servers") + async def codex_cookbook_servers(request: Request): + _scope_owner(request, COOKBOOK_READ_SCOPES) + state = _read_cookbook_state() + servers = state.get("env", {}).get("servers") or [] + # Strip ssh creds / passwords; keep only what's needed to pick a host. + cleaned = [] + for s in servers: + cleaned.append({ + "name": s.get("name"), + "host": s.get("host"), + "port": s.get("port"), + "env": s.get("env"), + "envPath": s.get("envPath"), + "platform": s.get("platform"), + "modelDirs": s.get("modelDirs"), + }) + return {"servers": cleaned} + + @router.get("/cookbook/output/{session_id}") + async def codex_cookbook_output(request: Request, session_id: str, tail: int = 400): + _scope_owner(request, COOKBOOK_READ_SCOPES) + # Defensive: session_id must be the tmux-style id we issue + # (`serve-XXXX` / `cookbook-XXXX` / `queue-XXXX`); anything else + # would let the agent run arbitrary `tmux capture-pane` targets. + import re as _re + if not _re.fullmatch(r"[a-zA-Z0-9_-]+", session_id): + raise HTTPException(400, "Invalid session id") + tail = max(20, min(int(tail or 400), 4000)) + # Resolve the task's host (if any) from cookbook state so we can + # ssh to the right box, exactly as the UI does in _reconnectTask. + state = _read_cookbook_state() + tasks = state.get("tasks") or [] + task = next((t for t in tasks if t.get("sessionId") == session_id), None) + if task is None: + raise HTTPException(404, "task not found") + host = (task.get("remoteHost") or "").strip() + ssh_port = (task.get("sshPort") or "").strip() + # Prefer the persisted log file over the tmux pane. The pane gets + # overwritten by the post-crash neofetch banner + bash prompt the + # moment vllm exits; the log file is the raw stdout/stderr and + # survives unchanged. Falls back to pane for older tasks predating + # the tee-to-log runner change. + log_path = f"/tmp/odysseus-tmux/{session_id}.log" + inner = ( + f"if [ -s {log_path} ]; then tail -n {tail} {log_path}; " + f"else tmux capture-pane -t {session_id} -p -S -{tail}; fi" + ) + if host: + port_flag = f"-p {ssh_port} " if ssh_port and ssh_port != "22" else "" + import shlex + cmd = f"ssh {port_flag}{host} {shlex.quote(inner)}" + else: + cmd = inner + result = await _run_shell(cmd, timeout=15) + return { + "session_id": session_id, + "host": host or "local", + "exit_code": result.get("exit_code"), + "output": result.get("stdout", ""), + "task": _redact_task(task), + } + + @router.post("/cookbook/serve") + async def codex_cookbook_serve(request: Request, body: dict[str, Any] = Body(default_factory=dict)): + _scope_owner(request, COOKBOOK_LAUNCH_SCOPES) + # Wraps /api/model/serve with the SAME validation the UI uses. + # _validate_serve_cmd (called inside model_serve) rejects shell + # metachars and requires the leading binary to be in the + # cookbook allowlist (vllm / python3 / sglang / llama-server / ...). + from routes.cookbook_helpers import ServeRequest + # Accept friendly aliases agents naturally reach for. Without these, + # passing `host` silently maps to nothing and the serve runs LOCAL + # instead of on the intended remote — exactly the bug an agent + # would never debug on its own. + norm = dict(body or {}) + if "host" in norm and "remote_host" not in norm: + norm["remote_host"] = norm.pop("host") + if "model" in norm and "repo_id" not in norm: + norm["repo_id"] = norm.pop("model") + if "ssh_port" not in norm and "port" in norm and (str(norm.get("port") or "").isdigit() and int(norm["port"]) >= 1000): + # Heuristic: if `port` looks like an SSH port (≥1000) and there's + # no explicit ssh_port, treat it as such. UI ports (8000, 8001, + # 30000) belong inside the cmd string, not here. + pass # leave as-is — user's `port` here is ambiguous; skip remap. + try: + req = ServeRequest(**norm) + except Exception as exc: + raise HTTPException(400, f"Invalid serve payload: {exc}") + serve_endpoint = _find_endpoint(None, "POST", "/api/model/serve") + # Fall back to importing from the cookbook router registered on app. + if serve_endpoint is None: + from fastapi import FastAPI + app: FastAPI = request.app + for route in app.routes: + if getattr(route, "path", None) == "/api/model/serve" and "POST" in getattr(route, "methods", set()): + serve_endpoint = route.endpoint + break + if serve_endpoint is None: + raise HTTPException(503, "model serve endpoint unavailable") + return await serve_endpoint(request, req) + + @router.post("/cookbook/stop/{session_id}") + async def codex_cookbook_stop(request: Request, session_id: str): + _scope_owner(request, COOKBOOK_LAUNCH_SCOPES) + import re as _re + if not _re.fullmatch(r"[a-zA-Z0-9_-]+", session_id): + raise HTTPException(400, "Invalid session id") + state = _read_cookbook_state() + tasks = state.get("tasks") or [] + task = next((t for t in tasks if t.get("sessionId") == session_id), None) + host = ((task or {}).get("remoteHost") or "").strip() + ssh_port = ((task or {}).get("sshPort") or "").strip() + if host: + port_flag = f"-p {ssh_port} " if ssh_port and ssh_port != "22" else "" + cmd = f"ssh {port_flag}{host} \"tmux kill-session -t {session_id}\"" + else: + cmd = f"tmux kill-session -t {session_id}" + result = await _run_shell(cmd, timeout=10) + return {"session_id": session_id, "exit_code": result.get("exit_code"), "host": host or "local"} + + @router.get("/cookbook/cached") + async def codex_cookbook_cached(request: Request, host: str | None = None): + """List cached models on a configured server (or local if host is omitted). + Mirrors `list_cached_models` from the chat agent so external agents have + the same inventory view before deciding what to serve/download.""" + _scope_owner(request, COOKBOOK_READ_SCOPES) + # Hit /api/model/cached internally, with the same modelDirs the chat + # agent's list_cached_models would resolve from cookbook state. + state = _read_cookbook_state() + env = state.get("env") if isinstance(state, dict) else {} + servers = (env.get("servers") if isinstance(env, dict) else None) or [] + HF_DEFAULTS = {"~/.cache/huggingface/hub", "~/.cache/huggingface"} + def _dirs_for(srv: dict) -> str: + mds = srv.get("modelDirs") if isinstance(srv, dict) else None + if isinstance(mds, list): + extras = [d for d in mds if isinstance(d, str) and d.strip() and d.strip() not in HF_DEFAULTS] + return ",".join(extras) + if isinstance(mds, str) and mds.strip() not in HF_DEFAULTS: + return mds + return "" + # Resolve friendly host name → real host (matches list_cached_models flow). + resolved_host = host or "" + srv: dict[str, Any] = {} + if host: + srv = next( + (s for s in servers if isinstance(s, dict) + and (s.get("name") == host or s.get("host") == host)), + {}, + ) + if srv and srv.get("host"): + resolved_host = srv["host"] + else: + srv = next((s for s in servers if isinstance(s, dict) and not (s.get("host") or "").strip()), {}) + params: dict[str, str] = {} + if resolved_host: + params["host"] = resolved_host + md = _dirs_for(srv) + if md: + params["model_dir"] = md + if srv.get("port"): + params["ssh_port"] = str(srv["port"]) + if srv.get("platform"): + params["platform"] = srv["platform"] + cached_endpoint = _find_endpoint(None, "GET", "/api/model/cached") + if cached_endpoint is None: + from fastapi import FastAPI + app: FastAPI = request.app + for route in app.routes: + if getattr(route, "path", None) == "/api/model/cached" and "GET" in getattr(route, "methods", set()): + cached_endpoint = route.endpoint + break + if cached_endpoint is None: + raise HTTPException(503, "model cached endpoint unavailable") + # The endpoint reads host/model_dir/ssh_port/platform as kwargs. + return await cached_endpoint( + request, + host=params.get("host") or None, + model_dir=params.get("model_dir") or None, + ssh_port=params.get("ssh_port") or None, + platform=params.get("platform") or None, + ) + + @router.get("/cookbook/presets") + async def codex_cookbook_presets(request: Request): + """List saved serve presets (model + host + port + launch cmd). + Counterpart to `list_serve_presets`. Use BEFORE composing a `serve` + body — the user's saved preset usually has the working cmd already.""" + _scope_owner(request, COOKBOOK_READ_SCOPES) + state = _read_cookbook_state() + presets = state.get("presets") or [] + out = [] + for p in presets: + if not isinstance(p, dict): + continue + out.append({ + "name": p.get("name"), + "model": p.get("model") or p.get("modelId"), + "host": p.get("host") or p.get("remoteHost"), + "port": p.get("port"), + "cmd": p.get("cmd"), + }) + return {"presets": out, "default_host": (state.get("env") or {}).get("defaultServer", "")} + + @router.post("/cookbook/preset/{name}") + async def codex_cookbook_serve_preset(request: Request, name: str): + """Launch a saved preset by name. Reuses the working cmd + host the + user already saved, avoiding the cmd-allowlist trial-and-error loop.""" + _scope_owner(request, COOKBOOK_LAUNCH_SCOPES) + import re as _re + if not _re.fullmatch(r"[A-Za-z0-9 _.:@\-]+", name): + raise HTTPException(400, "Invalid preset name") + state = _read_cookbook_state() + presets = state.get("presets") or [] + lname = name.lower().strip() + chosen = next( + (p for p in presets if isinstance(p, dict) and (p.get("name") or "").lower() == lname), + None, + ) + if chosen is None: + chosen = next( + (p for p in presets if isinstance(p, dict) and lname in (p.get("name") or "").lower()), + None, + ) + if chosen is None: + raise HTTPException(404, f"No preset matching {name!r}") + repo_id = chosen.get("model") or chosen.get("modelId") or "" + cmd = (chosen.get("cmd") or "").strip() + host = chosen.get("host") or chosen.get("remoteHost") or "" + if not repo_id or not cmd or cmd.startswith("(adopted"): + raise HTTPException(400, f"Preset {chosen.get('name')!r} has no launchable cmd " + "(adopted from external launch). Use POST /cookbook/serve " + "with the actual cmd instead.") + # Reuse the serve handler we already validated. + from routes.cookbook_helpers import ServeRequest + body = {"repo_id": repo_id, "cmd": cmd} + if host: + body["remote_host"] = host + try: + req = ServeRequest(**body) + except Exception as exc: + raise HTTPException(400, f"Preset payload invalid: {exc}") + serve_endpoint = _find_endpoint(None, "POST", "/api/model/serve") + if serve_endpoint is None: + from fastapi import FastAPI + app: FastAPI = request.app + for route in app.routes: + if getattr(route, "path", None) == "/api/model/serve" and "POST" in getattr(route, "methods", set()): + serve_endpoint = route.endpoint + break + if serve_endpoint is None: + raise HTTPException(503, "model serve endpoint unavailable") + return await serve_endpoint(request, req) + + @router.post("/cookbook/adopt") + async def codex_cookbook_adopt(request: Request, body: dict[str, Any] = Body(default_factory=dict)): + """Adopt an existing tmux session (one started via raw ssh+tmux) into + cookbook tracking. Needed when serve_model rejects a cmd and the + agent falls back to direct ssh — without adoption the session is + invisible to the UI. Body: {tmux_session, model, host?, port?}.""" + _scope_owner(request, COOKBOOK_LAUNCH_SCOPES) + norm = dict(body or {}) + sess = (norm.get("tmux_session") or norm.get("session_id") or "").strip() + model = (norm.get("model") or norm.get("repo_id") or "").strip() + host = (norm.get("host") or norm.get("remote_host") or "").strip() + port = norm.get("port") or 8000 + import re as _re + if not sess or not _re.fullmatch(r"[a-zA-Z0-9_-]+", sess): + raise HTTPException(400, "tmux_session required, [a-zA-Z0-9_-]+ only") + if not model: + raise HTTPException(400, "model required") + # Verify the tmux session exists on the target host before adopting. + import shlex + if host: + check = f"ssh {shlex.quote(host)} 'tmux has-session -t {shlex.quote(sess)}'" + else: + check = f"tmux has-session -t {shlex.quote(sess)}" + chk = await _run_shell(check, timeout=8) + if chk.get("exit_code") not in (0, None): + raise HTTPException(404, f"tmux session {sess!r} not found on {host or 'local'}") + # Write into cookbook_state.json. + import time as _t, json as _json + from core.atomic_io import atomic_write_json + from pathlib import Path as _Path + cookbook_state_path = _Path("/app/data/cookbook_state.json") + try: + state = _json.loads(cookbook_state_path.read_text(encoding="utf-8")) + except Exception: + state = {} + tasks = state.setdefault("tasks", []) + if any(isinstance(t, dict) and t.get("sessionId") == sess for t in tasks): + return {"ok": True, "already_tracked": True, "session_id": sess} + tasks.append({ + "id": sess, "sessionId": sess, + "name": model.split("/")[-1] if "/" in model else model, + "type": "serve", "status": "running", + "output": f"Adopted externally-launched session {sess!r} on {host or 'local'}.", + "ts": int(_t.time() * 1000), + "payload": {"repo_id": model, "remote_host": host, "_cmd": "(adopted — launched outside cookbook)", "port": int(port)}, + "remoteHost": host, "sshPort": "", "platform": "linux", + "_serveReady": False, "_endpointAdded": False, "_adoptedExternally": True, + }) + try: + atomic_write_json(cookbook_state_path, state) + except Exception as exc: + raise HTTPException(500, f"state write failed: {exc}") + return {"ok": True, "session_id": sess, "host": host or "local"} + return router diff --git a/routes/cookbook_helpers.py b/routes/cookbook_helpers.py index 9efb30d..454c67b 100644 --- a/routes/cookbook_helpers.py +++ b/routes/cookbook_helpers.py @@ -546,6 +546,13 @@ def _append_serve_preflight_exit_lines(runner_lines: list[str], *, keep_shell_op runner_lines.append('if [ -n "$ODYSSEUS_PREFLIGHT_EXIT" ]; then') runner_lines.append(' echo ""; echo "=== Process exited with code $ODYSSEUS_PREFLIGHT_EXIT ==="') if keep_shell_open: + # Decouple the post-crash interactive shell from the persistent log + # file. fds 3/4 were saved BEFORE the tee redirect at the top of + # the runner; restoring them here means the neofetch banner the + # user's .zshrc prints lands on the tmux pane only, not in the + # log file the agent's tail_serve_output reads. + runner_lines.append(' exec 1>&3 2>&4 3>&- 4>&- 2>/dev/null || true') + runner_lines.append(' sleep 0.2 # let tee child flush + exit') runner_lines.append(' exec "${SHELL:-/bin/bash}"') else: runner_lines.append(' exit "$ODYSSEUS_PREFLIGHT_EXIT"') @@ -563,7 +570,11 @@ def _append_serve_exit_code_lines( if is_pip_install: runner_lines.append('if [ $ODYSSEUS_CMD_EXIT -eq 0 ]; then echo ""; echo "DOWNLOAD_OK"; fi') if keep_shell_open: - runner_lines.append('echo ""; echo "=== Process exited with code $ODYSSEUS_CMD_EXIT ==="; exec "${SHELL:-/bin/bash}"') + runner_lines.append('echo ""; echo "=== Process exited with code $ODYSSEUS_CMD_EXIT ==="') + # See preflight branch above for the rationale on restoring fds 3/4. + runner_lines.append('exec 1>&3 2>&4 3>&- 4>&- 2>/dev/null || true') + runner_lines.append('sleep 0.2 # let tee child flush + exit') + runner_lines.append('exec "${SHELL:-/bin/bash}"') else: runner_lines.append('echo ""; echo "=== Process exited with code $ODYSSEUS_CMD_EXIT ==="') runner_lines.append('exit "$ODYSSEUS_CMD_EXIT"') diff --git a/routes/cookbook_routes.py b/routes/cookbook_routes.py index 56b95d6..bc15899 100644 --- a/routes/cookbook_routes.py +++ b/routes/cookbook_routes.py @@ -998,6 +998,21 @@ def setup_cookbook_routes() -> APIRouter: else: # ── Linux/Termux: bash + tmux (existing flow) ── runner_lines = ["#!/bin/bash"] + # Mirror every line of stdout+stderr into a persistent log file + # on the host running the serve. This is the file tail_serve_output + # reads when the tmux pane has been overwritten by the post-crash + # bash prompt — without it, the agent's diagnostic tool sees the + # neofetch banner instead of the actual Python traceback. + # We save the original fds to 3/4 so we can RESTORE them before + # `exec ${SHELL}` at the end of the script. Without that restore, + # the post-crash interactive shell's neofetch banner ALSO gets + # teed into the log file and `tail -N` returns ONLY the banner — + # the actual traceback ends up earlier than the tail window. + runner_lines.append("mkdir -p /tmp/odysseus-tmux 2>/dev/null || true") + runner_lines.append("exec 3>&1 4>&2") + runner_lines.append( + f"exec > >(tee -a /tmp/odysseus-tmux/{session_id}.log) 2>&1" + ) runner_lines.extend(_user_shell_path_bootstrap()) runner_lines.append('ODYSSEUS_PREFLIGHT_EXIT=""') # Put Odysseus's own venv bin on PATH (local runs only) so the serve @@ -1940,6 +1955,151 @@ def setup_cookbook_routes() -> APIRouter: return {"models": out} + # Rate-limit for the orphan-tmux adoption sweep. The UI polls + # tasks/status every ~3s; we don't want to SSH every host on every + # poll. 20s is fast enough that a model the agent launched in the + # background shows up "almost immediately" in the UI without being + # wasteful. + _last_orphan_sweep_ts = [0.0] + _ORPHAN_SWEEP_MIN_INTERVAL_S = 20.0 + + def _maybe_sweep_orphans(tasks: list, state: dict) -> None: + """Scan each configured cookbook server for `serve-*` tmux sessions + the cookbook doesn't know about and adopt them into state.tasks. + + Writes are conditional: if no orphans are found, nothing is touched. + Rate-limited so polling UIs don't trigger SSH on every refresh. + """ + import time as _time + import subprocess + logger.info(f"_maybe_sweep_orphans: entered, last_ts={_last_orphan_sweep_ts[0]}") + now = _time.monotonic() + if now - _last_orphan_sweep_ts[0] < _ORPHAN_SWEEP_MIN_INTERVAL_S: + logger.info(f"_maybe_sweep_orphans: rate-limited, {now - _last_orphan_sweep_ts[0]:.1f}s since last") + return + _last_orphan_sweep_ts[0] = now + + env = state.get("env") if isinstance(state, dict) else {} + servers = env.get("servers") if isinstance(env, dict) else [] + logger.info(f"orphan sweep starting: {len(servers) if isinstance(servers, list) else 0} server(s), known_sids={len([t for t in tasks if isinstance(t, dict) and t.get('sessionId')])}") + if not isinstance(servers, list): + return + + known_sids = { + t.get("sessionId") for t in tasks + if isinstance(t, dict) and t.get("sessionId") + } + + adopted_any = False + for srv in servers: + if not isinstance(srv, dict): + continue + host = (srv.get("host") or "").strip() + if not host: + continue # local-only entry; the /proc scan handles it + if not _REMOTE_HOST_RE.match(host): + continue + sport = str(srv.get("port") or "").strip() + ssh_base = ["ssh", "-o", "ConnectTimeout=4", "-o", "StrictHostKeyChecking=no"] + if sport and sport != "22": + if not _SSH_PORT_RE.match(sport): + continue + ssh_base.extend(["-p", sport]) + + try: + ls = subprocess.run( + ssh_base + [host, "tmux ls 2>/dev/null"], + timeout=6, capture_output=True, text=True, + ) + except Exception: + continue + for line in (ls.stdout or "").splitlines(): + sid = line.split(":", 1)[0].strip() + if not sid or not _SESSION_ID_RE.match(sid): + continue + # Only adopt sessions that LOOK like model serves; ignore + # bare numeric tmux sessions and unrelated work. + if not (sid.startswith("serve-") or sid.startswith("cookbook-")): + continue + if sid in known_sids: + continue + # Skip zombie / idle-shell sessions. A tmux session left + # over from a crashed vllm just shows a bash prompt — + # adopting it would pollute the UI with "running" tasks + # that aren't actually serving anything. pane_current_command + # is the foreground process in the pane right now; only + # real model serves leave a python/vllm/etc. process there. + try: + pc = subprocess.run( + ssh_base + [host, "tmux", "list-panes", "-t", sid, + "-F", "#{pane_current_command}"], + timeout=4, capture_output=True, text=True, + ) + cur = (pc.stdout or "").strip().splitlines() + except Exception: + cur = [] + LIVE_PROCS = {"python", "python3", "vllm", "llama-server", + "llama_cpp_main", "sglang", "lmdeploy", + "ollama", "node", "uvicorn"} + if not any(c in LIVE_PROCS for c in cur): + continue + # Try to recover a plausible repo_id + port from the + # pane buffer. Cheap heuristic — if we can't, register + # with placeholder fields; the UI still shows it. + try: + cap = subprocess.run( + ssh_base + [host, "tmux", "capture-pane", "-t", sid, "-p", "-S", "-300"], + timeout=6, capture_output=True, text=True, + ) + pane = cap.stdout or "" + except Exception: + pane = "" + import re as _re_orphan + # vLLM banner: "model /path/...". Falls back to the + # raw vllm-serve command if the banner already scrolled. + m_model = _re_orphan.search(r"model\s+(\S+)", pane) + model = m_model.group(1) if m_model else "" + if not model: + m_serve = _re_orphan.search(r"vllm\s+serve\s+(\S+)", pane) + model = m_serve.group(1) if m_serve else f"adopted:{sid}" + m_port = _re_orphan.search(r"--port\s+(\d+)", pane) + port = int(m_port.group(1)) if m_port else 0 + + import time as _t2 + tasks.append({ + "id": sid, + "sessionId": sid, + "name": model.split("/")[-1] if "/" in model else model, + "type": "serve", + "status": "running", + "output": f"Auto-adopted from orphan tmux session on {host}. " + "Open the task to see live output.", + "ts": int(_t2.time() * 1000), + "payload": { + "repo_id": model, + "remote_host": host, + "_cmd": "(orphan tmux session — original launch cmd unknown)", + "port": port, + }, + "remoteHost": host, + "sshPort": sport, + "platform": "linux", + "_serveReady": False, + "_endpointAdded": False, + "_adoptedExternally": True, + }) + known_sids.add(sid) + adopted_any = True + logger.info(f"auto-adopted orphan tmux session {sid!r} on {host}") + + if adopted_any: + try: + from core.atomic_io import atomic_write_json + state["tasks"] = tasks + atomic_write_json(_cookbook_state_path, state) + except Exception as e: + logger.warning(f"orphan sweep: state write failed: {e}") + @router.get("/api/cookbook/tasks/status") async def cookbook_tasks_status(request: Request): """Check status of all active cookbook tmux sessions. @@ -1993,6 +2153,7 @@ def setup_cookbook_routes() -> APIRouter: # Load saved tasks from cookbook state tasks = [] + state = {} if _cookbook_state_path.exists(): try: state = json.loads(_cookbook_state_path.read_text(encoding="utf-8")) @@ -2004,6 +2165,21 @@ def setup_cookbook_routes() -> APIRouter: except Exception: pass + # Orphan-tmux auto-adoption sweep. When the agent (or anyone) + # SSH-launches a `serve-*` tmux session — usually because + # serve_model rejected `source ... && vllm ...` or because of a + # manual relaunch via tmux send-keys — that session is invisible + # to the cookbook UI even though it's a live model server. The + # sweep finds those orphans on each configured remote host and + # writes them into state.tasks with _adoptedExternally=True, so + # they show up in the UI on the next poll without anyone having + # to remember to call adopt_served_model. Rate-limited via the + # module-level _last_orphan_sweep so we don't SSH every 3s. + try: + _maybe_sweep_orphans(tasks, state) + except Exception as _sweep_e: + logger.warning(f"orphan sweep failed (non-fatal): {_sweep_e!r}") + results = [] for task in tasks: session_id = task.get("sessionId", "") @@ -2063,7 +2239,12 @@ def setup_cookbook_routes() -> APIRouter: if _tport and _tport != "22": ssh_base.extend(["-p", str(_tport)]) check_cmd = ssh_base + [remote, "tmux", "has-session", "-t", session_id] - capture_cmd = ssh_base + [remote, "tmux", "capture-pane", "-t", session_id, "-p", "-S", "-50"] + # Capture 500 lines (was 50) so a Python traceback survives + # the post-crash neofetch banner + bash prompt that otherwise + # fills the visible tail. Without this, output_tail ends up + # as just "Locale: C / Ubuntu_Odysseus ❯" and the agent + # can't diagnose the actual error. + capture_cmd = ssh_base + [remote, "tmux", "capture-pane", "-t", session_id, "-p", "-S", "-500"] elif IS_WINDOWS: # LOCAL Windows task: launched as a detached process (no tmux). # Liveness comes from the .pid file, output from the @@ -2072,7 +2253,7 @@ def setup_cookbook_routes() -> APIRouter: capture_cmd = None else: check_cmd = ["tmux", "has-session", "-t", session_id] - capture_cmd = ["tmux", "capture-pane", "-t", session_id, "-p", "-S", "-50"] + capture_cmd = ["tmux", "capture-pane", "-t", session_id, "-p", "-S", "-500"] local_win_task = (not remote) and IS_WINDOWS diff --git a/src/agent_loop.py b/src/agent_loop.py index c0a7cc6..6dbdd6e 100644 --- a/src/agent_loop.py +++ b/src/agent_loop.py @@ -330,6 +330,7 @@ If the user asks for a reminder/alarm before the event, pass `reminder_minutes` "ui_control": "- ```ui_control``` — Control the UI: toggle tools on/off, OPEN PANELS, open email reply drafts, switch models, change themes. Commands: `toggle on/off` (names: bash/shell, web/search, research, incognito, document_editor/documents), `open_panel ` (panels: documents, gallery, email, sessions, notes, memories/brain, skills, settings, cookbook), `open_email_reply ` (opens an email compose document, does NOT send), `set_mode agent/chat`, `switch_model `, `set_theme `, `create_theme ` (optional key=val for advanced colors AND background effects: bgPattern=, bgEffectColor=#RRGGBB, bgEffectIntensity=, bgEffectSize=, frosted=true|false). \"open documents\" / \"open library\" / \"show gallery\" / \"open inbox\" / \"open notes\" / \"open cookbook\" all map to `open_panel `. Theme presets: dark, light, midnight, paper, cyberpunk, retrowave, forest, ocean, ume, copper, terminal, organs, lavender, gpt, claude, cute.", "list_served_models": "- ```list_served_models``` — Show what the Cookbook (LLM-serving subsystem) is currently running. NO args. Use this for ANY 'what's running' / 'what's serving' / 'show my cookbook' / 'is anything up' query. DO NOT shell out (`ps aux`, `docker ps`, etc.) — this tool is the source of truth. Failed serve tasks include recent logs plus diagnosis/retry suggestions; use those suggestions to call `serve_model` again with an adjusted command when appropriate.", "stop_served_model": "- ```stop_served_model``` — Stop a running model server. Args (JSON): {\"session_id\": \"\"}. Use for 'kill my cookbook' / 'stop the model' / 'shut down vLLM'.", + "tail_serve_output": "- ```tail_serve_output``` — Read the actual tmux stderr/traceback of a CURRENTLY failing cookbook task. Args (JSON): {\"session_id\": \"\", \"tail\": 150?}. **Use ONLY after** you just launched something via `serve_model` AND `list_served_models` reports YOUR new task as `crashed`/`error`. DO NOT use it on old stopped/completed download tasks (they're historical noise — won't predict whether a new launch succeeds). DO NOT call it before launching a fresh attempt. When you do call it, bump `tail` to 400+ only if the visible error references 'see root cause above'.", "download_model": "- ```download_model``` — Download a HuggingFace model. Args (JSON): {\"repo_id\": \"Qwen/Qwen3-8B\", \"host\": \"user@gpu-box\"?, \"include\": \"*Q4_K_M*\"?}.", "serve_model": "- ```serve_model``` — Start serving a model with vLLM / SGLang / llama.cpp / Ollama / Diffusers. Args (JSON): {\"repo_id\": \"...\", \"cmd\": \"vllm serve ... --port 8000\" or \"python3 -m sglang.launch_server ... --port 30000\" or \"python3 scripts/diffusion_server.py --model diffusers/stable-diffusion-xl-1.0-inpainting-0.1 --port 8100\", \"host\": \"user@gpu-box\"?}. For image/inpaint/diffusion models, use the `scripts/diffusion_server.py` command exactly. After launch, call `list_served_models`; if it returns a diagnosis with an adjusted command, retry with that command.", "list_downloads": "- ```list_downloads``` — Show in-progress HuggingFace model downloads (filters Cookbook tasks/status to downloads only). NO args. Use for 'what's downloading' / 'show my downloads' / 'check download progress'.", @@ -1646,6 +1647,28 @@ async def stream_agent_loop( _tool_type_counts: collections.Counter = collections.Counter() _THINK_RE = re.compile(r'.*?', re.DOTALL | re.IGNORECASE) _force_answer = False # set by loop-breaker → next round runs with NO tools + # Supervisor: how many times we've nudged the model after it announced + # an action without emitting the tool call. Capped to prevent a model + # that *can't* call the tool from looping forever. + _intent_nudge_count = 0 + _MAX_INTENT_NUDGES = 2 + + # "I said I would, then didn't" detector. The pattern that breaks debug + # loops on weak models (deepseek-v4-flash mid-2026): the model writes + # "Let me tail the output to see the error" and then ends the turn with + # no tool_calls. The intent is sincere but the function call gets dropped. + # Match the common phrasings + an action verb that maps to an available + # tool, so we don't nudge on harmless transitional text like "let me + # know what you think". + _INTENT_RE = re.compile( + r"(?:^|\n)\s*(?:let me|i'?ll|i will|going to|let's)\s+" + r"(?:tail|check|investigate|look at|see|tail|read|fetch|inspect|" + r"verify|diagnose|examine|debug|capture|grab|pull|view|run|call|" + r"trigger|launch|start|kick off|stop|kill|restart|adopt|serve|" + r"register|adopt|list|search|find|query|hit|ping|test)" + r"\b[^.\n]{0,140}", + re.IGNORECASE, + ) # Document streaming state (persists across rounds) _doc_acc = "" # accumulated tool-call JSON arguments @@ -1992,6 +2015,46 @@ async def stream_agent_loop( # never re-verify an unchanged state in a loop. _effectful_used = False continue + # ── Intent-without-action supervisor ───────────────────── + # Catch "Let me tail the output" / "I'll check the logs" / + # "Let me investigate" patterns where the model announces an + # action but emits no tool_call. The bug shows up most on + # smaller models trained to verbalize plans before acting. + # We inject one sharp nudge ("you said you would X — call the + # actual tool now") and loop again. Capped at + # _MAX_INTENT_NUDGES so a model that genuinely cannot use the + # tool doesn't pin us in a forever loop. + _intent_text = _THINK_RE.sub("", cleaned_round).strip() + _intent_match = _INTENT_RE.search(_intent_text) if _intent_text else None + # Only nudge when the round REALLY looks like an unfinished + # promise: short response (<400 chars), no fenced code/answer, + # and an action-intent phrase was matched. Long answers that + # happen to contain "let me know" are not stalls. + _looks_like_promise = ( + _intent_match is not None + and len(_intent_text) < 400 + and "```" not in _intent_text + and _intent_nudge_count < _MAX_INTENT_NUDGES + ) + if _looks_like_promise: + _intent_nudge_count += 1 + _matched_phrase = _intent_match.group(0).strip() + logger.info(f"[agent] intent-without-action nudge #{_intent_nudge_count} on round {round_num}: {_matched_phrase!r}") + messages.append({ + "role": "system", + "content": ( + f"You just wrote: \"{_matched_phrase}\" — but ended the " + "turn without making the actual tool call. The user can " + "see you announced the action but didn't run it, which " + "is the most frustrating thing you can do. " + "DO IT NOW: emit the actual function call this turn. " + "If you decided not to do it after all, say so plainly in " + "one sentence instead of restating the plan." + ), + }) + # Visible signal in the stream so the user knows we caught it. + yield f'data: {json.dumps({"type": "agent_step", "round": round_num + 1})}\n\n' + continue break # no tools — done # ── Loop-breaker (Terminus-style stall detector) ────────────── diff --git a/src/agent_tools.py b/src/agent_tools.py index 2785623..f162bc5 100644 --- a/src/agent_tools.py +++ b/src/agent_tools.py @@ -19,7 +19,7 @@ logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Constants (kept here — sub-modules import from here) # --------------------------------------------------------------------------- -MAX_AGENT_ROUNDS = 20 +MAX_AGENT_ROUNDS = 50 SHELL_TIMEOUT = 60 PYTHON_TIMEOUT = 30 MAX_OUTPUT_CHARS = 10_000 diff --git a/src/tool_execution.py b/src/tool_execution.py index b0e8e2d..c43fca9 100644 --- a/src/tool_execution.py +++ b/src/tool_execution.py @@ -700,6 +700,7 @@ async def execute_tool_block( do_manage_documents, do_manage_settings, do_manage_notes, do_manage_calendar, do_download_model, do_serve_model, do_list_served_models, do_stop_served_model, + do_tail_serve_output, do_list_downloads, do_cancel_download, do_search_hf_models, do_list_cached_models, do_list_serve_presets, do_serve_preset, do_adopt_served_model, do_list_cookbook_servers, @@ -864,6 +865,9 @@ async def execute_tool_block( elif tool == "stop_served_model": desc = "stop_served_model" result = await do_stop_served_model(content, owner=owner) + elif tool == "tail_serve_output": + desc = "tail_serve_output" + result = await do_tail_serve_output(content, owner=owner) elif tool == "list_downloads": desc = "list_downloads" result = await do_list_downloads(content, owner=owner) diff --git a/src/tool_implementations.py b/src/tool_implementations.py index c7b2649..c46a10c 100644 --- a/src/tool_implementations.py +++ b/src/tool_implementations.py @@ -5,6 +5,7 @@ Extracted tool implementation functions (do_* and helpers) from agent_tools.py. These handle the actual execution logic for each tool type. """ +import asyncio import json import logging import os @@ -2603,6 +2604,8 @@ async def _cookbook_env_for_host(host: str) -> Dict[str, Any]: return { "env_prefix": env_prefix, + "env_type": env_kind, + "env_path": env_path, "gpus": env_root.get("gpus") or "", "platform": platform, "hf_token": env_root.get("hfToken") or "", @@ -3041,6 +3044,31 @@ async def do_serve_model(content: str, owner: Optional[str] = None) -> Dict: # the UI uses. Without env_prefix, `vllm serve …` lands in a shell # without the user's venv and fails 'command not found'. env_cfg = await _cookbook_env_for_host(host) + # Rewrite bare `vllm` / `python3` leading tokens to the venv's absolute + # binary path when the target host has a venv configured. SSH non- + # interactive shells often leave ~/.local/bin ahead of the venv bin on + # PATH even with the venv activated, so `vllm serve` finds the wrong + # binary and crashes early (e.g. compute_89 torch ABI errors on an old + # user-site torch). This mirrors what static/js/cookbook.js does in + # _buildServeCmd for the UI launch path. + env_path = (env_cfg.get("env_path") or "").rstrip("/") + env_type = (env_cfg.get("env_type") or env_cfg.get("env") or "").lower() + if env_type == "venv" and env_path: + venv_bin = f"{env_path}/bin" + # Match the FIRST shell-token: skip leading KEY=VAL env-var prefixes + # (CUDA_VISIBLE_DEVICES=… VLLM_USE_FLASHINFER_SAMPLER=…) before the binary. + import re as _re3 + tokens = cmd.split() + idx = 0 + env_re = _re3.compile(r"^[A-Za-z_][A-Za-z0-9_]*=") + while idx < len(tokens) and env_re.match(tokens[idx]): + idx += 1 + if idx < len(tokens): + head = tokens[idx] + if head in ("vllm", "python3", "python"): + tokens[idx] = f"{venv_bin}/{head}" + cmd = " ".join(tokens) + payload["cmd"] = cmd if env_cfg.get("env_prefix"): payload["env_prefix"] = env_cfg["env_prefix"] if env_cfg.get("gpus"): payload["gpus"] = env_cfg["gpus"] if env_cfg.get("hf_token"): payload["hf_token"] = env_cfg["hf_token"] @@ -3059,7 +3087,19 @@ async def do_serve_model(content: str, owner: Optional[str] = None) -> Dict: ) note = "" if registered else " (state-write failed — task may not show in UI)" return {"output": f"Serving {repo_id} (session: {sid}){note}", "session_id": sid, "exit_code": 0} - return {"error": data.get("error", "Serve failed"), "exit_code": 1} + # FastAPI HTTPException puts the message under `detail`, not `error`. + # Surface BOTH so the agent sees "Invalid characters in cmd" (from + # _validate_serve_cmd rejecting `&&`/`source`/`cd`) instead of + # the generic "Serve failed", which leaves it with nothing to act on. + err_msg = data.get("error") or data.get("detail") or "Serve failed" + hint = "" + if isinstance(err_msg, str) and "cmd" in err_msg.lower(): + hint = (" — the cmd must START with an allowlisted binary " + "(vllm, python3, llama-server, ollama, sglang, lmdeploy, node, npx). " + "Do NOT prefix with `cd …`, `source …`, or chain with `&&`. " + "env_prefix (e.g. `source ~/qwen35-env/bin/activate`) is added " + "automatically from the host's saved venv settings.") + return {"error": f"{err_msg}{hint}", "exit_code": 1} except Exception as e: return {"error": str(e), "exit_code": 1} @@ -3103,13 +3143,31 @@ async def do_list_served_models(content: str, owner: Optional[str] = None) -> Di "exit_code": 0, } + # Sort so the agent sees what's actually LIVE first. Stopped/error/ + # completed tasks are mostly historical noise — they shouldn't lead + # the list when something is genuinely serving. + _ORDER = { + "ready": 0, "running": 1, "loading": 1, "warming": 1, + "queued": 2, "starting": 2, + "error": 5, "crashed": 5, "failed": 5, + "stopped": 6, "killed": 6, "cancelled": 6, "canceled": 6, + "done": 7, "completed": 7, "finished": 7, + } + def _rank(t: Dict[str, Any]) -> int: + phase = (t.get("phase") or t.get("status") or "unknown").lower() + return _ORDER.get(phase, 3) + merged.sort(key=_rank) + cb_n = len(cookbook_tasks) ext_n = len(external) + live_n = sum(1 for t in merged if _rank(t) <= 2) header = [] if cb_n: header.append(f"{cb_n} cookbook-tracked") if ext_n: header.append(f"{ext_n} external") + if live_n: + header.insert(0, f"{live_n} LIVE") lines = [f"Running: {', '.join(header)}."] for t in merged: phase = t.get("phase") or t.get("status", "unknown") @@ -3136,8 +3194,20 @@ async def do_list_served_models(content: str, owner: Optional[str] = None) -> Di if t.get("status") == "error" and t.get("output_tail"): tail = str(t.get("output_tail") or "").strip() if tail: + # Prefer a window around a Python traceback if one exists, + # falling back to the last 30 lines. The previous 6-line + # tail showed only the post-crash bash prompt / neofetch + # banner ("Locale: C / Ubuntu_Odysseus ❯") — useless for + # diagnosis. The traceback we want is usually 50-200 lines + # earlier in the buffer. + _tail_lines = tail.splitlines() + _shown = _tail_lines[-30:] + for _i, _ln in enumerate(_tail_lines): + if "Traceback (most recent call last)" in _ln or "ERROR" in _ln or "Error:" in _ln: + _shown = _tail_lines[_i:_i + 40] + break lines.append(" recent log:") - for line in tail.splitlines()[-6:]: + for line in _shown: lines.append(f" {line[:220]}") if t.get("external") and t.get("cmdline_preview"): lines.append(f" cmd: {t['cmdline_preview']}") @@ -3243,6 +3313,125 @@ async def do_stop_served_model(content: str, owner: Optional[str] = None) -> Dic ) +async def do_tail_serve_output(content: str, owner: Optional[str] = None) -> Dict: + """Capture the last N lines of a cookbook task's tmux pane — remote-aware. + + Used by the agent to debug a failed/stuck serve: list_served_models tells + you the task is `crashed`, this tool returns the actual stderr/traceback + so the agent can match it against a known fix (compute_89 nvcc mismatch, + flashinfer version mismatch, OOM, missing kernels, etc.) and decide + whether to relaunch via serve_model with new flags. + """ + import httpx + import shlex + try: + args = _parse_tool_args(content) + except ValueError: + return {"error": "Invalid JSON arguments", "exit_code": 1} + session_id = (args.get("session_id") or "").strip() + if not session_id: + return {"error": "session_id is required (from list_served_models)", "exit_code": 1} + import re as _re + if not _re.fullmatch(r"[a-zA-Z0-9_-]+", session_id): + return {"error": "Invalid session_id format", "exit_code": 1} + try: + tail = int(args.get("tail") or 400) + except (TypeError, ValueError): + tail = 400 + tail = max(20, min(tail, 4000)) + headers = _internal_headers() + remote = (args.get("remote_host") or args.get("host") or "").strip() + sport = (args.get("ssh_port") or "").strip() + # Resolve host from cookbook state if caller didn't pass one — same + # lookup _cookbook_kill_session uses. + if not remote: + state: Dict[str, Any] = {} + try: + async with httpx.AsyncClient(timeout=10) as client: + resp = await client.get(f"{_COOKBOOK_BASE}/api/cookbook/state", headers=headers) + state = resp.json() or {} + except Exception as e: + logger.debug(f"cookbook state lookup failed for {session_id}: {e}") + if isinstance(state, dict): + for t in (state.get("tasks") or []): + if isinstance(t, dict) and (t.get("sessionId") == session_id or t.get("id") == session_id): + remote = t.get("remoteHost") or "" + if not sport: + sport = t.get("sshPort") or "" + break + # Prefer the persisted /tmp/odysseus-tmux/SESSION.log file over the + # live tmux pane. The pane is what the user would see scrolling on + # their screen — including the post-crash neofetch banner and the + # idle bash prompt that overwrites the actual traceback the moment + # vllm exits. The log file is the raw stdout/stderr of the wrapped + # process and survives the crash unchanged. We only fall back to + # the pane when the log file doesn't exist (older sessions launched + # before the tmux+tee wrapper was added). + log_path = f"/tmp/odysseus-tmux/{session_id}.log" + pane_inner = f"tmux capture-pane -t {shlex.quote(session_id)} -p -S -{tail} 2>/dev/null" + file_inner = f"tail -n {tail} {shlex.quote(log_path)} 2>/dev/null" + inner = ( + f"if [ -s {shlex.quote(log_path)} ]; then {file_inner}; " + f"else {pane_inner}; fi" + ) + if remote: + _pf = f"-p {shlex.quote(str(sport))} " if sport and str(sport) != "22" else "" + cmd = ( + f"ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no " + f"{_pf}{shlex.quote(remote)} {shlex.quote(inner)}" + ) + host_label = remote + else: + cmd = inner + host_label = "local" + try: + async with httpx.AsyncClient(timeout=20) as client: + resp = await client.post(f"{_COOKBOOK_BASE}/api/shell/exec", + json={"command": cmd}, headers=headers) + if resp.status_code >= 400: + return {"error": f"shell/exec returned HTTP {resp.status_code}: {resp.text[:200]}", "exit_code": 1} + data = resp.json() if resp.content else {} + output_text = (data.get("stdout") or "").strip() + stderr_text = (data.get("stderr") or "").strip() + rc = data.get("exit_code") + if rc not in (None, 0) and not output_text: + already_gone = any(s in (stderr_text or "").lower() for s in ("no server running", "can't find session", "session not found")) + if already_gone: + return {"output": f"Tmux session {session_id} on {host_label} is gone (task already exited).", "exit_code": 0, "session_id": session_id, "host": host_label} + return {"error": f"capture-pane failed on {host_label}: {stderr_text or f'exit {rc}'}", "exit_code": 1} + # Dedupe download-progress noise. A 100-shard HF download produces + # tens of thousands of `model-NN-of-MM.safetensors: 91%|...` lines + # that all look the same to the agent and drown the actual error. + # Keep only one sample per (file, decile-percent) bucket. + import re as _re2 + lines = output_text.splitlines() + dedup_lines = [] + seen_progress = set() + progress_re = _re2.compile(r"^([\w./\-]+):\s+(\d+)%") + for ln in lines: + m = progress_re.match(ln.strip()) + if m: + key = (m.group(1), int(m.group(2)) // 10) # bucket by 10% + if key in seen_progress: + continue + seen_progress.add(key) + dedup_lines.append(ln) + output_text = "\n".join(dedup_lines) + # Hard cap so the agent doesn't blow its token budget. + MAX_CHARS = 8000 + if len(output_text) > MAX_CHARS: + output_text = "…(earlier output truncated)…\n" + output_text[-MAX_CHARS:] + return { + "output": output_text or "(empty pane)", + "session_id": session_id, + "host": host_label, + "tail_lines": tail, + "exit_code": 0, + } + except Exception as e: + return {"error": str(e), "exit_code": 1} + + async def do_list_downloads(content: str, owner: Optional[str] = None) -> Dict: """List in-flight model downloads (filters /api/cookbook/tasks/status to type=download).""" import httpx @@ -3615,38 +3804,133 @@ async def do_serve_preset(content: str, owner: Optional[str] = None) -> Dict: async def do_list_cached_models(content: str, owner: Optional[str] = None) -> Dict: - """List models already cached locally (or on a remote host).""" + """List models already cached locally and/or on remote hosts. + + With no `host` arg, scans EVERY configured Cookbook server (and local) + and aggregates — so the agent sees the full inventory in one call + instead of having to query each server individually. + """ import httpx try: args = _parse_tool_args(content) if content.strip() else {} except ValueError: return {"error": "Invalid JSON arguments", "exit_code": 1} - params: Dict[str, str] = {} raw_host = (args.get("host") or "").strip() - host = await _resolve_cookbook_host(raw_host) if raw_host else "" - if host: - params["host"] = host - if args.get("model_dir"): - params["model_dir"] = args["model_dir"] - if args.get("ssh_port"): - params["ssh_port"] = str(args["ssh_port"]) - if args.get("platform"): - params["platform"] = args["platform"] + headers = _internal_headers() + + async def _scan_one(host_label: str, host_val: str, ssh_port: str = "", + platform: str = "", model_dir: str = "") -> list: + """Hit /api/model/cached for one host; tag each returned model with its source.""" + p: Dict[str, str] = {} + if host_val: + p["host"] = host_val + # Caller-provided override beats per-server config beats nothing. + if args.get("model_dir"): + p["model_dir"] = args["model_dir"] + elif model_dir: + p["model_dir"] = model_dir + if ssh_port: + p["ssh_port"] = ssh_port + elif args.get("ssh_port"): + p["ssh_port"] = str(args["ssh_port"]) + if platform: + p["platform"] = platform + elif args.get("platform"): + p["platform"] = args["platform"] + try: + async with httpx.AsyncClient(timeout=60) as client: + resp = await client.get(f"{_COOKBOOK_BASE}/api/model/cached", + params=p, headers=headers) + data = resp.json() + ms = data.get("models", []) if isinstance(data, dict) else (data or []) + for m in ms: + m["host"] = host_label or "local" + return ms or [] + except Exception as e: + logger.debug(f"list_cached_models scan({host_label}) failed: {e}") + return [] + + # When the caller specifies a host explicitly, scan only that one (old behaviour). + # Otherwise iterate every configured server + local so the agent doesn't + # have to repeat the call per server. try: - async with httpx.AsyncClient(timeout=60) as client: - resp = await client.get(f"{_COOKBOOK_BASE}/api/model/cached", - params=params, headers=_internal_headers()) - data = resp.json() - models = data.get("models", []) if isinstance(data, dict) else data + # Pull configured servers from cookbook state (used for resolving + # modelDirs both when caller specifies a host and when we scan all). + servers: list = [] + try: + async with httpx.AsyncClient(timeout=10) as client: + st = await client.get(f"{_COOKBOOK_BASE}/api/cookbook/state", headers=headers) + st_data = st.json() if st.headers.get("content-type", "").startswith("application/json") else {} + servers = (st_data.get("env", {}) or {}).get("servers") or [] + except Exception as e: + logger.debug(f"server list fetch failed: {e}") + st_data = {} + + def _dirs_for(server_record: Dict[str, Any]) -> str: + """Comma-joined modelDirs from a saved server record (Settings). + + Filters out the HF cache (~/.cache/huggingface/hub) — the backend + scan script always scans it by default, so re-passing it as an + extra model_dir is redundant AND confuses some path-handling + edge cases where the extra dir suppresses the deeper scan. + We only need to forward the NON-default dirs (e.g. /mnt/HADES/models). + """ + mds = server_record.get("modelDirs") if isinstance(server_record, dict) else None + HF_DEFAULTS = {"~/.cache/huggingface/hub", "~/.cache/huggingface"} + if isinstance(mds, list): + extras = [d for d in mds if isinstance(d, str) and d.strip() and d.strip() not in HF_DEFAULTS] + return ",".join(extras) + if isinstance(mds, str) and mds.strip() not in HF_DEFAULTS: + return mds + return "" + + if raw_host: + host = await _resolve_cookbook_host(raw_host) + # Find this host's saved record so its modelDirs apply too. + srv = next( + (s for s in servers if isinstance(s, dict) + and (s.get("name") == raw_host or s.get("host") == host or s.get("host") == raw_host)), + {}, + ) + models = await _scan_one(raw_host, host, model_dir=_dirs_for(srv)) + else: + # Always include local. Local's saved record is the one with no host. + local_srv = next((s for s in servers if isinstance(s, dict) and not (s.get("host") or "").strip()), {}) + scans: list = [_scan_one("local", "", model_dir=_dirs_for(local_srv))] + for s in servers: + if not isinstance(s, dict): + continue + name = s.get("name") or s.get("host") + host_val = s.get("host") or "" + if not host_val: + continue + scans.append(_scan_one( + name, + host_val, + ssh_port=str(s.get("port") or ""), + platform=s.get("platform") or "", + model_dir=_dirs_for(s), + )) + results = await asyncio.gather(*scans, return_exceptions=False) + # Dedupe by (host, repo_id) — same model could appear in both HF cache + Ollama list. + seen = set() + models: list = [] + for batch in results: + for m in batch: + key = (m.get("host", ""), m.get("repo_id", "")) + if key in seen: + continue + seen.add(key) + models.append(m) if not models: - # Filesystem cache scans can miss models downloaded into the HF - # default cache when the server has no explicit model_dir configured. - # Still surface completed Cookbook downloads so the agent doesn't - # incorrectly assume a model is absent and re-download it. + # Cache scans can miss models downloaded into the HF default cache + # when the server has no explicit model_dir configured. Surface + # completed Cookbook download tasks so the agent doesn't conclude + # a model is absent and re-download it. downloaded = [] try: async with httpx.AsyncClient(timeout=10) as client: - st = await client.get(f"{_COOKBOOK_BASE}/api/cookbook/state", headers=_internal_headers()) + st = await client.get(f"{_COOKBOOK_BASE}/api/cookbook/state", headers=headers) state = st.json() if st.headers.get("content-type", "").startswith("application/json") else {} for t in (state.get("tasks") or []): if not isinstance(t, dict) or t.get("type") != "download": @@ -3654,27 +3938,44 @@ async def do_list_cached_models(content: str, owner: Optional[str] = None) -> Di if (t.get("status") or "").lower() not in {"done", "completed"}: continue task_host = t.get("remoteHost") or (t.get("payload") or {}).get("remote_host") or "" - if host and task_host != host: + if raw_host and task_host != raw_host: continue repo = t.get("modelId") or t.get("repoId") or (t.get("payload") or {}).get("repo_id") or t.get("name") if repo and repo not in downloaded: downloaded.append(repo) except Exception: downloaded = [] + host_str = f" on {raw_host}" if raw_host else "" if downloaded: - host_str = f" on {raw_host or host}" if (raw_host or host) else "" lines = [f"No cache paths were detected{host_str}, but Cookbook has completed download task(s):"] lines.extend(f"- {repo} — downloaded via Cookbook task" for repo in downloaded) return {"output": "\n".join(lines), "models": [{"repo_id": repo, "source": "cookbook_task"} for repo in downloaded], "exit_code": 0} - host_str = f" on {raw_host or host}" if (raw_host or host) else "" return {"output": f"No cached models found{host_str}.", "exit_code": 0} - lines = [f"{len(models)} cached model(s):"] - for m in models: - name = m.get("repo_id", "?") - sz = m.get("size") or (f"{m.get('size_bytes', 0) / (1024**3):.1f}GB" if m.get("size_bytes") else "") - inc = " (incomplete)" if m.get("has_incomplete") else "" - kind = " [diffusion]" if m.get("is_diffusion") else "" - lines.append(f"- {name}{kind} — {sz}{inc}") + # Multi-host scan: group by host so the agent sees inventory per server. + # Single-host scan: flat list (matches old output shape). + if raw_host: + lines = [f"{len(models)} cached model(s) on {raw_host}:"] + for m in models: + name = m.get("repo_id", "?") + sz = m.get("size") or (f"{m.get('size_bytes', 0) / (1024**3):.1f}GB" if m.get("size_bytes") else "") + inc = " (incomplete)" if m.get("has_incomplete") else "" + kind = " [diffusion]" if m.get("is_diffusion") else "" + lines.append(f"- {name}{kind} — {sz}{inc}") + else: + from collections import defaultdict as _dd + by_host = _dd(list) + for m in models: + by_host[m.get("host", "local")].append(m) + lines = [f"{len(models)} cached model(s) across {len(by_host)} server(s):"] + for host_name in sorted(by_host.keys()): + lines.append(f"\n[{host_name}]") + for m in by_host[host_name]: + name = m.get("repo_id", "?") + sz = m.get("size") or (f"{m.get('size_bytes', 0) / (1024**3):.1f}GB" if m.get("size_bytes") else "") + inc = " (incomplete)" if m.get("has_incomplete") else "" + kind = " [diffusion]" if m.get("is_diffusion") else "" + backend = f" ({m.get('backend')})" if m.get("backend") else "" + lines.append(f"- {name}{kind}{backend} — {sz}{inc}") return {"output": "\n".join(lines), "models": models, "exit_code": 0} except Exception as e: return {"error": str(e), "exit_code": 1} diff --git a/src/tool_index.py b/src/tool_index.py index 506e55d..3fecb79 100644 --- a/src/tool_index.py +++ b/src/tool_index.py @@ -31,7 +31,15 @@ ALWAYS_AVAILABLE = frozenset({ # keyword hints when the user is actually talking about cookbook. # Keeping the always-on set small leaves room in the ~16-tool # budget for manage_tasks / manage_calendar / etc. - "list_served_models", "stop_served_model", + "list_served_models", "stop_served_model", "tail_serve_output", + # Serving is a core agent capability — keep these always available so + # the router doesn't lose them on phrasings like "servic" / "fire up" / "boot". + "serve_model", "serve_preset", "list_serve_presets", + "list_cached_models", "list_cookbook_servers", + # Fallback when serve_model's allowlist rejects a cmd or when the + # model was launched out-of-band via bash+tmux — without this the + # session is invisible to the cookbook UI even though it's running. + "adopt_served_model", # Generic API loopback — the catch-all when no named tool fits. "app_api", }) @@ -104,9 +112,10 @@ BUILTIN_TOOL_DESCRIPTIONS: Dict[str, str] = { "manage_notes": "Create and manage notes and checklists (Google Keep-style). ALWAYS use this for note/todo/checklist/reminder creation — NEVER hit /api/notes via app_api. Accepts natural-language `due_date` like 'tomorrow at 9am' or '11pm today' (parsed in the USER'S timezone). The due_date IS the reminder — it fires a notification at that time, so do NOT also create a calendar event for the same reminder. Set colors, labels, pin, archive. Do NOT use manage_memory for note content.", "manage_calendar": "Calendar event management: list, create, update, delete. Each event can carry a tag/category (event_type — work/personal/health/travel/meal/social/admin/other) and importance (low/normal/high/critical). Use ISO datetimes; supports all-day events. For event reminders/alarms, pass reminder_minutes; this creates the Notes reminder, so do not also call manage_notes for the same reminder.", "download_model": "Download a HuggingFace model to a local or remote server. Specify repo_id (e.g. 'Qwen/Qwen3-8B'), optional server host, and optional include filter for specific files.", - "serve_model": "Start serving a model with vLLM, SGLang, llama.cpp, Ollama, or Diffusers. For image/inpainting/diffusion use python3 scripts/diffusion_server.py --model --port 8100. After launch, call list_served_models for readiness/errors and retry suggestions.", + "serve_model": "Start serving a model with vLLM, SGLang, llama.cpp, Ollama, or Diffusers. cmd MUST start with the binary directly — e.g. `vllm serve /mnt/HADES/models/Qwen3.5-397B-A17B-AWQ --port 8003 --tensor-parallel-size 8 …`. NEVER prefix with `cd …`, `source …`, or chain with `&&`/`||` — those get rejected by the validator. The venv activation (env_prefix) and CUDA env are added automatically from the target host's saved settings. For image/inpainting/diffusion use python3 scripts/diffusion_server.py --model --port 8100. After launch, call list_served_models for readiness/errors and retry suggestions. If serve_model fails with 'Invalid characters in cmd', simplify to the bare binary + args.", "list_served_models": "List currently running model servers in the Cookbook — shows status (loading, ready, idle, error), model name, port, throughput, and serve failure diagnosis/retry suggestions. Use when the user asks 'what's running', 'show my cookbook', 'which models are up', 'what's serving'.", "stop_served_model": "Stop a running model server in the Cookbook by session ID or model name. Use when the user says 'kill my cookbook', 'stop the model', 'kill the serve', 'shut down vLLM', 'cancel the running model'.", + "tail_serve_output": "Read the actual tmux stderr/traceback of a cookbook serve/download task. Use to debug WHY a task is `crashed`/`error` (compute_89 nvcc mismatch, OOM, missing kernels, wrong attention backend, etc.) so you can call serve_model with adjusted flags. Pass session_id from list_served_models; tail defaults to 300, bump if the error references 'see root cause above'.", "list_downloads": "List in-progress HuggingFace model downloads in the Cookbook. Shows model name, phase, percent, session ID. Use for 'what's downloading', 'show my downloads', 'check download progress'.", "cancel_download": "Cancel an in-progress model download by tmux session ID. Use for 'cancel the download', 'stop downloading X', 'kill the download'. Call list_downloads first to get the session_id.", "search_hf_models": "Search HuggingFace for models matching a query (e.g. 'qwen 8B', 'flux', 'llama-3 instruct'). Returns ranked repo IDs with sizes and download counts. Use for 'find a model', 'search huggingface for X', 'what models are there for Y'.", diff --git a/src/tool_schemas.py b/src/tool_schemas.py index f55fb82..ef0939c 100644 --- a/src/tool_schemas.py +++ b/src/tool_schemas.py @@ -721,6 +721,21 @@ FUNCTION_TOOL_SCHEMAS = [ } } }, + { + "type": "function", + "function": { + "name": "tail_serve_output", + "description": "Read the last N lines of a cookbook serve/download task's tmux pane. Use ONLY in this exact sequence: (1) the user asked to serve a model, (2) you launched it via serve_model, (3) list_served_models reports the NEW task as crashed/error, (4) call tail_serve_output on the new sessionId to find the root cause, (5) call serve_model again with adjusted flags. DO NOT call this on old stopped/completed download tasks — they are historical and won't tell you anything about the current attempt. DO NOT investigate past failures before launching; the environment may have changed since.", + "parameters": { + "type": "object", + "properties": { + "session_id": {"type": "string", "description": "Tmux session id from list_served_models (e.g. 'serve-abc12345', 'cookbook-a1b2c3d4')."}, + "tail": {"type": "integer", "description": "How many lines of pane scrollback to fetch (default 300, max 4000). Bump this if the error in the visible tail references an earlier line ('see root cause above')."}, + }, + "required": ["session_id"] + } + } + }, { "type": "function", "function": { diff --git a/static/js/cookbook-diagnosis.js b/static/js/cookbook-diagnosis.js index 4d6e807..ec81aa0 100644 --- a/static/js/cookbook-diagnosis.js +++ b/static/js/cookbook-diagnosis.js @@ -378,16 +378,12 @@ export const ERROR_PATTERNS = [ message: 'Model architecture too new for installed vLLM/transformers.', fixes: [ { label: 'Try --trust-remote-code', action: (panel) => _serveAutoRetry(panel, '--trust-remote-code'), autofix: true }, - { label: 'Update vLLM on server', action: (panel) => { - const taskEl = panel.closest('.cookbook-task'); - const task = taskEl ? _loadTasks().find(t => t.sessionId === taskEl.dataset.taskId) : null; - const host = task?.remoteHost || ''; - const prefix = _buildEnvPrefix(); - const pipCmd = prefix ? prefix + ' pip install -U vllm transformers' : 'pip install -U vllm transformers'; - const cmd = host ? _sshCmd(host, pipCmd) : pipCmd; - // Run in tmux so it doesn't timeout - const name = 'update-vllm'; - _launchServeTask(name, 'pip-update', cmd); + { label: 'Update vLLM on server', action: () => { + // Use the venv's python3 by absolute path when configured (SSH non- + // interactive sessions often pick user-site Python over the venv). + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('update-vllm', 'pip-update', `${_vp} -m pip install -U vllm transformers`); }}, ], }, @@ -395,16 +391,10 @@ export const ERROR_PATTERNS = [ pattern: /Either a revision or a version must be specified|transformers\.integrations\.hub_kernels|kernels\/layer/i, message: 'Transformers/kernels package mismatch.', fixes: [ - { label: 'Repair kernel package', action: (panel) => { - const taskEl = panel.closest('.cookbook-task'); - const task = taskEl ? _loadTasks().find(t => t.sessionId === taskEl.dataset.taskId) : null; - const host = task?.remoteHost || ''; - const prefix = _buildEnvPrefix(); - const pipCmd = prefix - ? prefix + ' python3 -m pip install --user --break-system-packages "kernels<0.15"' - : 'python3 -m pip install --user --break-system-packages "kernels<0.15"'; - const cmd = host ? _sshCmd(host, pipCmd) : pipCmd; - _launchServeTask('repair-kernels', 'pip-update', cmd); + { label: 'Repair kernel package', action: () => { + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('repair-kernels', 'pip-update', `${_vp} -m pip install --user --break-system-packages kernels<0.15`); }}, { label: 'Open Dependencies', action: () => _openCookbookDependencies('sglang') }, ], @@ -445,14 +435,10 @@ export const ERROR_PATTERNS = [ pattern: /Triton kernels.*Failed to import|cannot import name '\w+' from 'triton_kernels/i, message: 'Triton kernels version mismatch. Non-fatal warning — model will still run, just without optimized MoE kernels.', fixes: [ - { label: 'Update triton on server', action: (panel) => { - const taskEl = panel.closest('.cookbook-task'); - const task = taskEl ? _loadTasks().find(t => t.sessionId === taskEl.dataset.taskId) : null; - const host = task?.remoteHost || ''; - const prefix = _buildEnvPrefix(); - const pipCmd = prefix ? prefix + ' pip install -U triton triton-kernels' : 'pip install -U triton triton-kernels'; - const cmd = host ? _sshCmd(host, pipCmd) : pipCmd; - _launchServeTask('update-triton', 'pip-update', cmd); + { label: 'Update triton on server', action: () => { + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('update-triton', 'pip-update', `${_vp} -m pip install -U triton triton-kernels`); }}, ], }, @@ -474,14 +460,56 @@ export const ERROR_PATTERNS = [ pattern: /attention_sink|sliding.window.*not supported|sliding_window.*incompatible/i, message: 'Model uses attention features unsupported in this vLLM version.', fixes: [ - { label: 'Update vLLM on server', action: (panel) => { - const taskEl = panel.closest('.cookbook-task'); - const task = taskEl ? _loadTasks().find(t => t.sessionId === taskEl.dataset.taskId) : null; - const host = task?.remoteHost || ''; - const prefix = _buildEnvPrefix(); - const pipCmd = prefix ? prefix + ' pip install -U vllm' : 'pip install -U vllm'; - const cmd = host ? _sshCmd(host, pipCmd) : pipCmd; - _launchServeTask('update-vllm', 'pip-update', cmd); + { label: 'Update vLLM on server', action: () => { + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('update-vllm', 'pip-update', `${_vp} -m pip install -U vllm`); + }}, + ], + }, + { + // FlashInfer JIT-compiles attention kernels for the host GPU on first + // use. If the system /usr/bin/nvcc is older than CUDA 11.8 it can't + // target sm_89/sm_90 (Ada/Hopper), and the engine workers die before + // they can report a useful traceback. Two quick paths out: pick a + // non-flashinfer attention backend, or set CUDACXX to a newer nvcc + // (vLLM installs nvidia-cuda-nvcc into the venv — point at that). + pattern: /nvcc fatal\s+:\s+Unsupported gpu architecture 'compute_\d+'/i, + message: 'FlashInfer is JIT-compiling sampling kernels with an nvcc too old for this GPU (no sm_89 / sm_90 support — pre-CUDA 11.8). Changing the attention backend does not help — flashinfer JITs the SAMPLER too. The clean fix is to set VLLM_USE_FLASHINFER_SAMPLER=0 so vLLM uses its native sampler instead.', + suggestion: 'Suggested action: relaunch with VLLM_USE_FLASHINFER_SAMPLER=0 prepended. (Confirmed on the QuantTrio/Qwen3.5 model card as the canonical workaround.)', + fixes: [ + { label: 'Retry with VLLM_USE_FLASHINFER_SAMPLER=0', action: (panel) => _serveAutoRetryReplace(panel, '', 'VLLM_USE_FLASHINFER_SAMPLER=0 ', { prepend: true }) }, + { label: 'Uninstall flashinfer-python', action: () => { + // Hard fallback: vLLM 0.22 reaches into flashinfer for sampling kernels + // even with VLLM_USE_FLASHINFER_SAMPLER=0 in some configs. Removing + // the package forces it onto the native sampler. + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('uninstall-flashinfer', 'pip-update', `${_vp} -m pip uninstall flashinfer-python -y`); + }}, + { label: 'Edit serve', action: (panel) => _openServeEditFromDiagnosis(panel) }, + ], + }, + { + // vLLM <-> torch ABI mismatch: vLLM imports torch.library helpers + // (`infer_schema`, `register_fake`, etc.) that only exist on newer torch + // versions. When the installed torch is older, the import fails before + // any server code runs. Fix is to reinstall vllm (which pulls a matching + // torch) or upgrade torch directly. + pattern: /ImportError: cannot import name '[^']+' from 'torch(\.\w+)+'/i, + message: 'vLLM was built against a newer torch than what is installed. Reinstall vLLM so pip pulls a compatible torch (or upgrade torch directly).', + fixes: [ + { label: 'Reinstall vLLM (pulls matching torch)', action: () => { + // Absolute path to the venv's python3 — bare `python3` lands in the + // wrong site-packages over SSH when ~/.local/bin precedes the venv. + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('reinstall-vllm', 'pip-reinstall', `${_vp} -m pip install --force-reinstall vllm`); + }}, + { label: 'Upgrade torch only', action: () => { + const _vp = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` : 'python3'; + _launchServeTask('upgrade-torch', 'pip-update', `${_vp} -m pip install -U torch`); }}, ], }, @@ -607,59 +635,24 @@ export function _showDiagnosis(panel, diagnosis, sourceText) { }; if (fixes.length) { + // Always render fixes as inline buttons. The old "Actions ▾" dropdown + // (for >3 fixes) was broken — the menu wouldn't open in some panels and + // hid useful actions behind a non-working affordance. Inline buttons wrap + // naturally in `.cookbook-diag-fixes` (flex-wrap) so a long list reflows + // onto multiple rows instead of getting collapsed. const row = document.createElement('div'); row.className = 'cookbook-diag-fixes'; - - if (fixes.length <= 3) { - for (const fix of fixes) { - const btn = document.createElement('button'); - btn.className = 'cookbook-btn cookbook-diag-btn'; - btn.type = 'button'; - btn.innerHTML = _diagFixIcon(fix.label) + '' + _diagEsc(fix.label) + ''; - btn.addEventListener('click', (e) => { - e.stopPropagation(); - runFix(fix, btn); - }); - row.appendChild(btn); - } - body.appendChild(row); - return; - } - - const wrap = document.createElement('div'); - wrap.className = 'cookbook-diag-actions'; - - const trigger = document.createElement('button'); - trigger.className = 'cookbook-btn cookbook-diag-action-trigger'; - trigger.type = 'button'; - trigger.textContent = 'Actions'; - trigger.appendChild(document.createTextNode(' ▾')); - wrap.appendChild(trigger); - - const menu = document.createElement('div'); - menu.className = 'dropdown cookbook-diag-menu hidden'; for (const fix of fixes) { - const item = document.createElement('button'); - item.type = 'button'; - item.innerHTML = _diagFixIcon(fix.label) + '' + _diagEsc(fix.label) + ''; - item.addEventListener('click', async (e) => { + const btn = document.createElement('button'); + btn.className = 'cookbook-btn cookbook-diag-btn'; + btn.type = 'button'; + btn.innerHTML = _diagFixIcon(fix.label) + '' + _diagEsc(fix.label) + ''; + btn.addEventListener('click', (e) => { e.stopPropagation(); - if (item.dataset.busy || trigger.dataset.busy) return; - item.dataset.busy = '1'; - await runFix(fix, trigger, fix.label, () => menu.classList.add('hidden'), () => delete item.dataset.busy); + runFix(fix, btn); }); - menu.appendChild(item); + row.appendChild(btn); } - wrap.appendChild(menu); - trigger.addEventListener('click', (e) => { - e.stopPropagation(); - if (trigger.dataset.busy) return; - document.querySelectorAll('.cookbook-diag-menu').forEach(m => { - if (m !== menu) m.classList.add('hidden'); - }); - menu.classList.toggle('hidden'); - }); - row.appendChild(wrap); body.appendChild(row); } } diff --git a/static/js/cookbook.js b/static/js/cookbook.js index 507777a..358d664 100644 --- a/static/js/cookbook.js +++ b/static/js/cookbook.js @@ -353,6 +353,15 @@ function _buildEnvPrefixWindows() { } export function _buildServeCmd(f, modelName, backend) { + // When a venv is configured on the chosen server, use the venv's binaries + // by absolute path. Bare `vllm` / `python3` relies on PATH, and SSH non- + // interactive sessions often leave a user-site install (~/.local/bin/vllm) + // ahead of the venv's bin, so the WRONG vllm gets launched even with the + // venv activated. Absolute path sidesteps the whole PATH question. + const _isVenv = _envState.env === 'venv' && _envState.envPath; + const _venvBin = _isVenv ? (_envState.envPath.replace(/\/+$/, '') + '/bin/') : ''; + const _vllmBin = _venvBin ? `${_venvBin}vllm` : 'vllm'; + const _py3Bin = _venvBin ? `${_venvBin}python3` : 'python3'; let cmd = ''; if (backend === 'vllm') { const gpuId = f.gpu_id?.trim() || ''; @@ -361,7 +370,15 @@ export function _buildServeCmd(f, modelName, backend) { const _opts = _detectModelOptimizations(modelName); if (_opts.envVars.length) cmd += _opts.envVars.join(' ') + ' '; } - cmd += `vllm serve ${modelName} --host 0.0.0.0 --port ${f.port || '8000'}`; + // Pinned attention backend (Attention field). Empty = let vLLM pick. + const _attn = (f.vllm_attn_backend ?? '').toString().trim(); + if (_attn) cmd += `VLLM_ATTENTION_BACKEND=${_attn} `; + // Free-text "Env" field — verbatim KEY=VAL pairs (space-separated). + // Collapse any pasted newlines/tabs so the backend allowlist (which + // rejects \n / \r) doesn't trip on a multi-line paste from a model card. + const _extraEnv = (f.extra_env ?? '').toString().replace(/\s+/g, ' ').trim(); + if (_extraEnv) cmd += _extraEnv + ' '; + cmd += `${_vllmBin} serve ${modelName} --host 0.0.0.0 --port ${f.port || '8000'}`; cmd += ` --tensor-parallel-size ${f.tp || '1'}`; cmd += ` --max-model-len ${f.ctx || '8192'}`; cmd += ` --gpu-memory-utilization ${f.gpu_mem || '0.90'}`; @@ -389,7 +406,9 @@ export function _buildServeCmd(f, modelName, backend) { } else if (backend === 'sglang') { const gpuId = f.gpu_id?.trim() || ''; if (gpuId) cmd += `CUDA_VISIBLE_DEVICES=${gpuId} `; - cmd += `python3 -m sglang.launch_server --model-path ${modelName} --host 0.0.0.0 --port ${f.port || '30000'}`; + const _extraEnv = (f.extra_env ?? '').toString().replace(/\s+/g, ' ').trim(); + if (_extraEnv) cmd += _extraEnv + ' '; + cmd += `${_py3Bin} -m sglang.launch_server --model-path ${modelName} --host 0.0.0.0 --port ${f.port || '30000'}`; if (f.tp && f.tp !== '1') cmd += ` --tp ${f.tp}`; if (f.ctx) cmd += ` --context-length ${f.ctx}`; if (f.gpu_mem && f.gpu_mem !== '0.90') cmd += ` --mem-fraction-static ${f.gpu_mem}`; @@ -642,13 +661,20 @@ async function _fetchDependencies() { const winBlocked = !isLocal && _isWindows() && _winUnsupported.has(pkg.name); const note = pkg.status_note ? `
${esc(pkg.status_note)}
` : ''; const updateNote = pkg.installed && pkg.pip_update_available === false && pkg.update_note ? `
${esc(pkg.update_note)}
` : ''; - // Inline "Rebuild" tag for the llama_cpp row only. Styled as a - // .cookbook-dep-tag so it matches the LLM category tag's pill look, - // and lives to the LEFT of the category tag (clear affordance before - // the row "value"). - const _rebuildBtn = (pkg.name === 'llama_cpp') - ? `` - : ''; + // Inline rebuild/reinstall tag. Styled as a .cookbook-dep-tag so it + // matches the LLM category tag's pill look, and lives to the LEFT of the + // category tag. llama_cpp uses the /api/cookbook/rebuild-engine flow + // (clear cached binary so next serve recompiles); vllm/sglang use the + // diagnosis-style `_launchServeTask` with `pip install --force-reinstall` + // so the user can watch the pip install in the Running tab. + let _rebuildBtn = ''; + if (pkg.name === 'llama_cpp') { + _rebuildBtn = ``; + } else if (pkg.name === 'vllm' && pkg.installed) { + _rebuildBtn = ``; + } else if (pkg.name === 'sglang' && pkg.installed) { + _rebuildBtn = ``; + } return `
` + `
` + `
${esc(pkg.name)}
` @@ -696,7 +722,18 @@ async function _fetchDependencies() { // for PEP-668-locked system pythons (Arch, newer Debian). const _inEnv = _envState.env === 'venv' || _envState.env === 'conda'; const _pipFlags = (!_isWindows() && !_inEnv) ? ' --user --break-system-packages' : ''; - const _py = _isWindows() ? 'python' : 'python3'; + // Use the venv's python3 by absolute path when configured. Even with the + // env_prefix sourcing activate, SSH non-interactive sessions sometimes + // pick a `python3` ahead of the venv's bin on PATH, so the install + // silently lands in the wrong site-packages. + let _py; + if (_isWindows()) { + _py = 'python'; + } else if (_envState.env === 'venv' && _envState.envPath) { + _py = `${_envState.envPath.replace(/\/+$/, '')}/bin/python3`; + } else { + _py = 'python3'; + } const cmd = `${_py} -m pip install${upgrade ? ' -U' : ''}${_pipFlags} "${pipName}"`; let envPrefix = ''; if (_isWindows()) { @@ -1072,6 +1109,32 @@ function _wireTabEvents(body) { }); } + // "Reinstall" buttons for pip-based serving stacks (vllm, sglang). The + // deps list renders ASYNCHRONOUSLY after _fetchDependencies resolves, so + // attaching listeners directly here would miss buttons that don't exist + // yet. Use document-level delegation instead — the click always finds the + // right .cookbook-dep-reinstall button no matter when it was painted. + if (!document._cookbookReinstallWired) { + document._cookbookReinstallWired = true; + document.addEventListener('click', async (ev) => { + const btn = ev.target.closest?.('.cookbook-dep-reinstall'); + if (!btn) return; + const pkg = btn.dataset.reinstallPkg || ''; + if (!pkg) return; + ev.preventDefault(); + ev.stopPropagation(); + const sel = document.getElementById('hwfit-deps-server'); + if (sel) _applyServerSelection(sel.value); + const host = _envState.remoteHost || ''; + const where = host || 'this server'; + if (!confirm(`Reinstall ${pkg} on ${where}?\n\nRuns "pip install --force-reinstall --no-deps ${pkg}" as a tmux task. Watch progress in the Running tab.`)) return; + const _venvPy = (_envState.env === 'venv' && _envState.envPath) + ? `${_envState.envPath.replace(/\/+$/, '')}/bin/python3` + : 'python3'; + _launchServeTask(`reinstall-${pkg}`, 'pip-reinstall', `${_venvPy} -m pip install --force-reinstall --no-deps ${pkg}`); + }, true); + } + // Serve sort const serveSort = document.getElementById('serve-sort'); if (serveSort) { diff --git a/static/js/cookbookRunning.js b/static/js/cookbookRunning.js index 5f75a49..b532602 100644 --- a/static/js/cookbookRunning.js +++ b/static/js/cookbookRunning.js @@ -124,6 +124,14 @@ async function _openDownloadForGgufTask(task) { function _terminalServeDiagnosis(task, outputText) { const out = String(outputText || task?.output || ''); if (!task || task.type !== 'serve' || !['stopped', 'error', 'crashed', 'failed'].includes(task.status) || !out.trim()) return null; + // Pip tasks (Reinstall vLLM, Upgrade torch, etc.) ride on the serve task + // type so they get a tmux session + show up in Running tab — but they are + // NOT serve invocations. Their output is pip's own; the generic + // "Serve stopped before the model became reachable" message + Edit-serve + // fix make no sense. Bail so the panel just shows pip's output. + const _isPipTask = ((task.payload?.repo_id || '').startsWith('pip-')) + || /python3? -m pip\b/.test(task.payload?._cmd || ''); + if (_isPipTask) return null; if (_serveTaskLooksAwqOnLocalBackend(task, out)) { return { message: 'AWQ/GPTQ/FP8 cannot be served through llama.cpp/Ollama unified-memory mode.', @@ -249,7 +257,7 @@ const SERVE_STATE_KEY = 'cookbook-serve-state'; // Polling / timeout intervals const TASK_POLL_INTERVAL_MS = 3000; // delay between reconnect-loop iterations -const BG_MONITOR_INTERVAL_MS = 10000; // background task status poll +const BG_MONITOR_INTERVAL_MS = 5000; // background task status poll const STALE_PROGRESS_MS = 5 * 60 * 1000; // download with no progress this long = stale const STARTUP_STALE_PROGRESS_MS = 45 * 1000; // 0%-forever startup stall: retry much sooner @@ -523,6 +531,26 @@ function _serveOutputLooksReady(task) { function _normalizeTaskForDisplay(task) { if (!task || typeof task !== 'object') return task; + // Pip tasks (Reinstall vLLM / Upgrade torch / etc.) ride on the serve task + // type so they get tmux + the Running tab. They are NOT serves — their + // "ready" markers are pip's `Successfully installed` / `Requirement already + // satisfied`, not "Application startup complete". + const _isPipTask = ((task.payload?.repo_id || '').startsWith('pip-')) + || /python3? -m pip\b/.test(task.payload?._cmd || ''); + if (_isPipTask) { + // Override stale status: any pip task whose output carries pip's own + // success markers gets displayed as `done` regardless of what's in + // localStorage. Old pre-fix runs landed in error/stopped state and + // stuck there even after we taught the rest of the flow about pip + // tasks — this is the catch-all that flips them to Finished on render. + const out = String(task.output || ''); + const ranOk = /Successfully installed|Requirement already (?:satisfied|up-to-date)/i.test(out) + && !/error:|ERROR:/.test(out.slice(-1024)); + if (ranOk && task.status !== 'done' && task.status !== 'running') { + return { ...task, status: 'done' }; + } + return task; + } if (task.type === 'serve' && task.status === 'done' && !_serveOutputLooksReady(task)) { return { ...task, status: 'error' }; } @@ -2409,7 +2437,7 @@ async function _reconnectTask(el, task) { if (data.exit_code !== 0) { failCount++; if (failCount < 5) { - await new Promise(r => setTimeout(r, 5000)); + await new Promise(r => setTimeout(r, 3000)); continue; } try { @@ -2430,7 +2458,15 @@ async function _reconnectTask(el, task) { } const lastOutput = output.textContent || ''; - const diag = _diagnose(lastOutput); + // Pip tasks (Reinstall vLLM / Upgrade torch / etc.) must skip the + // generic serve `_diagnose` step. Their output is pip's own and the + // error patterns there (torch ABI traceback, "No module named torch", + // etc.) are routinely matched against the previous tmux scrollback, + // tagging a clean pip success as a crashed serve. Detection is the + // same shape as the looksSuccessful branch below. + const _isPipTaskDiag = ((task.payload?.repo_id || '').startsWith('pip-')) + || /python3? -m pip\b/.test(task.payload?._cmd || ''); + const diag = _isPipTaskDiag ? null : _diagnose(lastOutput); if (diag) { let diagEl = el.querySelector('.cookbook-diagnosis'); if (!diagEl) { @@ -2447,14 +2483,40 @@ async function _reconnectTask(el, task) { } else { const downloadLooksSuccessful = !lastOutput.includes('DOWNLOAD_FAILED') && (lastOutput.includes('DONE') || lastOutput.includes('100%') || lastOutput.includes('/snapshots/') || lastOutput.includes('Download complete') || lastOutput.includes('DOWNLOAD_OK')); + // Pip install / reinstall tasks are launched via _launchServeTask (so + // they show up in the Running tab + use tmux) but they aren't real + // serves — the cmd is `python3 -m pip ...` and the success markers + // are pip's own. Without this branch, a successful reinstall ends + // with no "Uvicorn running on" line and gets mis-flagged as a crashed + // serve. + const _isPipTask = ((task.payload?.repo_id || '').startsWith('pip-')) + || /python3? -m pip\b/.test(task.payload?._cmd || ''); + const pipLooksSuccessful = _isPipTask + && /Successfully installed|Requirement already (?:satisfied|up-to-date)/i.test(lastOutput) + && !/error:|ERROR:/.test(lastOutput.slice(-1024)); const serveLooksReady = task.type === 'serve' && _serveOutputLooksReady({ ...task, output: lastOutput }); - const looksSuccessful = task.type === 'download' ? downloadLooksSuccessful : serveLooksReady; + const looksSuccessful = task.type === 'download' + ? downloadLooksSuccessful + : (_isPipTask ? pipLooksSuccessful : serveLooksReady); if (!lastOutput.trim() || !looksSuccessful) { _updateTask(task.sessionId, { status: 'crashed' }); el.dataset.status = 'crashed'; const badge = el.querySelector('.cookbook-task-status'); if (badge) { badge.textContent = _statusLabel('crashed', task.type); badge.className = 'cookbook-task-status cookbook-task-crashed'; } - if (task.type === 'serve') { + if (_isPipTask) { + // Pip tasks: don't run the serve diagnosis (which would yell + // "Serve stopped before the model became reachable"). Show a + // pip-tailored message; the user can read pip's own error output + // directly above. + const _ranOk = /Successfully installed|Requirement already (?:satisfied|up-to-date)/i.test(lastOutput); + if (!_ranOk) { + _showDiagnosis(el, { + message: 'Pip install did not finish with a success marker. Check the output for the underlying error.', + suggestion: 'Suggested action: copy the troubleshooting bundle. Common causes: missing build deps, network blip, mismatched torch ABI.', + fixes: [], + }, lastOutput); + } + } else if (task.type === 'serve') { const diag = _diagnose(lastOutput) || { message: _serveTaskLooksAwqOnLocalBackend(task, lastOutput) ? 'AWQ/GPTQ/FP8 cannot be served through llama.cpp/Ollama unified-memory mode.' @@ -2533,6 +2595,28 @@ async function _reconnectTask(el, task) { } _showCookbookNotif(true); } else { + // Strong completion markers — `DOWNLOAD_OK` is emitted by our + // downloader wrapper AFTER the model snapshot is on disk, and + // `/snapshots/` only appears once HF has resolved the cached + // tree. Either is conclusive. Finalize as done immediately, skip + // the 30s debounce — the debounce only exists to guard against + // ambiguous markers (bare "100%" / "Download complete") which can + // appear mid-stream during multi-file downloads. + const _strongDone = task.type === 'download' + && (lastOutput.includes('DOWNLOAD_OK') || lastOutput.includes('/snapshots/')); + if (_strongDone) { + _updateTask(task.sessionId, { status: 'done', _doneConfirmAt: null, _lastStatusFlipAt: Date.now() }); + el.dataset.status = 'done'; + const badge = el.querySelector('.cookbook-task-status'); + if (badge) { badge.textContent = _statusLabel('done', task.type); badge.className = 'cookbook-task-status cookbook-task-done'; } + const _chk = el.querySelector('.cookbook-task-check'); if (_chk) _chk.style.display = ''; + const _sb = el.querySelector('.cookbook-task-serve-btn'); if (_sb) _sb.style.display = ''; + _showCookbookNotif(); + _refreshDepsAfterInstall(task); + _renderRunningTab(); + _processQueue(); + break; + } // Debounce the done flip. Tmux capture-pane can fail transiently // (network blip, ssh reconnect), and the verify has-session right // above can briefly report dead even when the session is in the @@ -2559,7 +2643,7 @@ async function _reconnectTask(el, task) { stillAlive = pData.exit_code === 0; } catch { /* network blip — treat as inconclusive, prefer running */ stillAlive = true; } if (stillAlive) { - _updateTask(task.sessionId, { status: 'running', _doneConfirmAt: null }); + _updateTask(task.sessionId, { status: 'running', _doneConfirmAt: null, _lastStatusFlipAt: Date.now() }); const _el = document.querySelector(`.cookbook-task[data-task-id="${task.sessionId}"]`); if (_el) { _el.dataset.status = 'running'; @@ -2571,7 +2655,7 @@ async function _reconnectTask(el, task) { } return; } - _updateTask(task.sessionId, { status: 'done', _doneConfirmAt: null }); + _updateTask(task.sessionId, { status: 'done', _doneConfirmAt: null, _lastStatusFlipAt: Date.now() }); const _el = document.querySelector(`.cookbook-task[data-task-id="${task.sessionId}"]`); if (_el) { _el.dataset.status = 'done'; @@ -2596,8 +2680,14 @@ async function _reconnectTask(el, task) { const snapshot = (data.stdout || '').trim(); if (snapshot) { + // Only auto-scroll to bottom if the user was already there. When + // they've scrolled up to read earlier output, leave their position + // alone so a fresh snapshot doesn't yank them back to the tail. + // 40px tolerance covers sub-pixel rounding + the moment between + // releasing the scrollbar and the next poll arriving. + const _atBottom = (output.scrollHeight - output.scrollTop - output.clientHeight) < 40; output.textContent = snapshot; - output.scrollTop = output.scrollHeight; + if (_atBottom) output.scrollTop = output.scrollHeight; // Live status parsing for download tasks if (task.type === 'download') { @@ -3153,16 +3243,27 @@ export async function _selfHealStaleTasks(opts = {}) { // itself fires every 10s, so this almost always fires too, but the // guard keeps a fast manual call from doubling up). const now = Date.now(); - if (now - _selfHealLastTs < 8000) return; + if (now - _selfHealLastTs < 4000) return; _selfHealLastTs = now; } const tasks = _loadTasks(); - const candidates = tasks.filter(t => - t.type === 'download' - && ['done', 'error', 'crashed', 'stopped'].includes(t.status) - && t.sessionId - && !String(t.sessionId).startsWith('queue-') - ); + const candidates = tasks.filter(t => { + if (t.type !== 'download') return false; + if (!['done', 'error', 'crashed', 'stopped'].includes(t.status)) return false; + if (!t.sessionId || String(t.sessionId).startsWith('queue-')) return false; + // Finished downloads with strong completion markers (DOWNLOAD_OK or HF + // /snapshots/ resolution) are demonstrably done — do not flip them back + // to running just because the tmux session is still alive (e.g., a + // long-lived shell that hosted the download or a flapping SSH that + // reports the session as up). This was the main source of finished↔ + // downloading oscillation on a flaky connection. + if (t.status === 'done' && /DOWNLOAD_OK|\/snapshots\//.test(t.output || '')) return false; + // Cooldown: never flip the same task more than once every 45s. A flapping + // SSH connection used to drive the badge back-and-forth on every probe + // cycle; this enforces a stable view between flaps. + if (t._lastStatusFlipAt && (Date.now() - t._lastStatusFlipAt < 45000)) return false; + return true; + }); if (!candidates.length) return; let flipped = 0; for (const t of candidates) { @@ -3180,6 +3281,7 @@ export async function _selfHealStaleTasks(opts = {}) { if (ft && ft.status !== 'running') { ft.status = 'running'; ft._selfHealed = true; + ft._lastStatusFlipAt = Date.now(); _saveTasks(fresh); flipped++; const _el = document.querySelector(`.cookbook-task[data-task-id="${t.sessionId}"]`); diff --git a/static/js/cookbookServe.js b/static/js/cookbookServe.js index 1ef503f..20d9305 100644 --- a/static/js/cookbookServe.js +++ b/static/js/cookbookServe.js @@ -613,6 +613,20 @@ function _rerenderCachedModels() { panelHtml += ``; panelHtml += ``; panelHtml += ``; + // Attention backend selector — pin the kernel impl. Default `auto` lets + // vLLM pick FlashInfer (which JITs on first use and breaks on older + // system nvcc) → FlashAttention → xformers. Forcing FLASH_ATTN skips + // the JIT entirely, fixing the `nvcc fatal: Unsupported gpu + // architecture 'compute_89'` failure mode on Ada / Hopper hosts. + const vllmAttnBackendOpts = ['auto', 'FLASH_ATTN', 'XFORMERS', 'FLASHINFER', 'TORCH_SDPA'] + .map(b => ``).join(''); + panelHtml += ``; + // Free-text env-vars field. Anything pasted here is prepended to the + // launch command verbatim. Use for CUDACXX, PATH overrides, NCCL_* + // tuning, or any other KEY=VALUE pair that doesn't have a dedicated + // field. After the venv activate runs, $VIRTUAL_ENV / $PATH / etc. are + // already exported so they expand correctly here. + panelHtml += ``; panelHtml += `
`; // Row 2b: Diffusers settings const diffDtypeOpts = ['bfloat16','float16','float32'].map(d => ``).join(''); @@ -1643,6 +1657,35 @@ function _rerenderCachedModels() { // Launch button panel.querySelector('.hwfit-serve-launch').addEventListener('click', async (ev) => { const _launchBtn = ev.currentTarget; + // Immediate visual feedback. The GPU probe + backend-warning prompt + // below can take ~1-2s before the task UI shows up, leaving the + // button looking dead. Drop in the same whirlpool spinner the rest of + // the cookbook uses (Probe GPUs, dependency installs, etc.) right + // away; restored on any early-return / failure path below. + const _origBtnHtml = _launchBtn.innerHTML; + const _origBtnDisabled = _launchBtn.disabled; + let _launchingWp = null; + const _restoreLaunchBtn = () => { + try { _launchingWp?.destroy?.(); } catch {} + _launchingWp = null; + _launchBtn.innerHTML = _origBtnHtml; + _launchBtn.disabled = _origBtnDisabled; + }; + _launchBtn.disabled = true; + _launchBtn.innerHTML = ''; + const _launchingWrap = document.createElement('span'); + _launchingWrap.className = 'hwfit-serve-launching'; + _launchingWrap.style.cssText = 'display:inline-flex;align-items:center;gap:6px;'; + _launchingWp = spinnerModule.createWhirlpool(18); + if (_launchingWp?.element) { + _launchingWp.element.style.margin = '0'; + _launchingWp.element.style.transform = 'translateY(-2px)'; + _launchingWrap.appendChild(_launchingWp.element); + } + const _launchingLabel = document.createElement('span'); + _launchingLabel.textContent = 'Launching…'; + _launchingWrap.appendChild(_launchingLabel); + _launchBtn.appendChild(_launchingWrap); // Final safety net: never launch with ctx beyond the model's trained // limit (or the absolute sanity ceiling when the limit is unknown). A // stale preset or typo (e.g. 16000000) overflows and, with a quantized @@ -1650,7 +1693,14 @@ function _rerenderCachedModels() { // command (then we respect their literal text). if (!_cmdManuallyEdited) _clampCtx(true); if (!_cmdManuallyEdited) updateCmd(); - const launchCmd = _cmdTextarea ? _cmdTextarea.value.trim() : panel._cmd; + // Pasted commands often carry hidden newlines / CRs / tabs from copies + // out of model cards or wrapped help text. The backend cmd allowlist + // rejects \n / \r outright (`Invalid characters in cmd`), so collapse + // all whitespace to single spaces before launch — same effect as the + // user manually re-flowing the textarea, no behavior change. + const _rawLaunchCmd = _cmdTextarea ? _cmdTextarea.value : panel._cmd; + const launchCmd = String(_rawLaunchCmd || '').replace(/\s+/g, ' ').trim(); + if (_cmdTextarea && _cmdTextarea.value !== launchCmd) _cmdTextarea.value = launchCmd; const serveState = {}; panel.querySelectorAll('.hwfit-sf').forEach(el => { if (el.type === 'checkbox') serveState[el.dataset.field] = el.checked; @@ -1659,6 +1709,7 @@ function _rerenderCachedModels() { serveState.backend = serveState.backend || (_detectBackend(m).backend) || 'vllm'; const backendWarning = _serveBackendWarning(m, repo, serveState.backend, serveState); if (backendWarning) { + _restoreLaunchBtn(); await window.styledConfirm(backendWarning.body, { title: backendWarning.title, confirmText: 'Edit settings', @@ -1689,7 +1740,7 @@ function _rerenderCachedModels() { `No GPU detected on ${_probeHost ? _probeHost : 'this host'}. ${serveState.backend.toUpperCase()} needs a visible CUDA/ROCm accelerator to start — launching now will most likely crash early.\n\nLaunch anyway?`, { title: 'No GPU detected', confirmText: 'Launch anyway', cancelText: 'Cancel', danger: true }, ); - if (!_proceed) return; + if (!_proceed) { _restoreLaunchBtn(); return; } } } catch { // Network / probe failure — don't block. Better to let the launch diff --git a/static/js/settings.js b/static/js/settings.js index dd9240f..c56f658 100644 --- a/static/js/settings.js +++ b/static/js/settings.js @@ -4566,6 +4566,8 @@ async function initUnifiedIntegrations() { { key: 'calendar:write', label: 'Calendar write', detail: 'Create and update calendar events' }, { key: 'memory:read', label: 'Memory', detail: 'Read memory when enabled' }, { key: 'memory:write', label: 'Memory write', detail: 'Write memory when enabled' }, + { key: 'cookbook:read', label: 'Cookbook', detail: 'List cookbook tasks + tail their tmux output (debug a model serve from outside the UI)' }, + { key: 'cookbook:launch', label: 'Cookbook launch', detail: 'Launch and stop cookbook serve tasks. Powerful: runs SSH commands on your configured servers, bounded by the same allowlist the UI uses (vllm/python3/sglang/llama-server/...)' }, ]; // Strict name-prefix match keeps Codex and Claude tokens in their own forms. const agentTokens = (Array.isArray(tokens) ? tokens : []).filter(tok => @@ -4578,6 +4580,7 @@ async function initUnifiedIntegrations() { email: '', calendar: '', memory: '', + cookbook: '', }; const _scopeNiceLabel = (label) => label.replace(/\s+(write|drafts?|send)$/i, ''); const _scopeAction = (key) => (key.split(':')[1] || '').toLowerCase(); diff --git a/static/style.css b/static/style.css index fcb607f..0671539 100644 --- a/static/style.css +++ b/static/style.css @@ -19281,6 +19281,11 @@ body.gallery-selecting .gallery-dl-btn, background: color-mix(in srgb, var(--fg) 7%, transparent); font-size: 12px; border-bottom: 1px solid color-mix(in srgb, var(--fg) 6%, transparent); + /* Pin the row so flex parents + Firefox mobile can't squeeze its height to 0, + which hides the type pill + model name and leaves only the sub-line + + output visible. */ + flex-shrink: 0; + min-height: 32px; } .cookbook-task-type { text-transform: uppercase;