merge: update adsb degraded branch

docs/agent-handoff.md

@@ -1,18 +1,489 @@
 # Agent Handoff
 
-## Current Release Goal
+Last updated: 2026-05-17
 
-Source branch: `codex/production-intelligence-terminal`
+## Repository State
 
-Registry image:
+Project: Crucix fork / Intelligence Terminal
+
+Local workspace:
+
+```text
+C:\Users\MrSphay\Documents\Codex\Crucix\intelligence-terminal
+```
+
+Remotes:
+
+```text
+origin   https://git.wilkensxl.de/MrSphay/intelligence-terminal.git
+upstream https://github.com/calesthio/Crucix.git
+```
+
+Current branch tip:
+
+```text
+Run `git rev-parse HEAD` after clone/pull. This handoff was updated by the `docs: sync issue tracker and handoff` commit after the implementation commit below.
+```
+
+Latest implementation commit before issue-sync documentation:
+
+```text
+53470cc701ec322080a89d220aef449b25850590
+```
+
+Both pushed branches currently point to this commit:
+
+```text
+origin/codex/production-intelligence-terminal
+origin/main
+```
+
+Gitea repository:
+
+```text
+https://git.wilkensxl.de/MrSphay/intelligence-terminal
+```
+
+Default branch observed through the Gitea API:
+
+```text
+codex/production-intelligence-terminal
+```
+
+## Agent Kit Requirements Applied
+
+The mandatory kit was cloned and reviewed first:
+
+```text
+C:\Users\MrSphay\Documents\Codex\Crucix\agent-kit
+```
+
+Rules applied from the kit:
+
+- Keep agent context in source control: `AGENTS.md`, `.codex/project.md`, and this handoff file.
+- Use Gitea Ubuntu runners for heavy verification and package publishing.
+- Keep Docker/Dockge operation first-class.
+- Do not commit secrets, `.env`, private logs, tokens, or generated `runs/` data.
+- Add report-only maintenance workflows for security, dependency checks, repo cleanup, release dry runs, and template compliance.
+- Poll pushed Gitea Actions until terminal state when a token is available.
+
+## What Was Implemented
+
+### Docker And Runtime
+
+- Docker image is Docker-first and Dockge/Pangolin suitable.
+- Browser auto-open is disabled by default through `AUTO_OPEN_BROWSER=false`.
+- Runtime health checks now work in the container without `wget` or host browser tools.
+- `runs` is persisted through a volume.
+- A later fix added `docker-entrypoint.sh` to prepare `/app/runs` before dropping privileges, so mounted volumes work with the non-root Node runtime.
+- `docker-compose.yml` uses the Gitea Registry image by default:
 
 ```text
 git.wilkensxl.de/mrsphay/intelligence-terminal:latest
 ```
 
-## Notes
+### API And Health
 
-- The repository is Docker-first and should stay suitable for Dockge/Pangolin.
-- Use `.env.example` as the operator-facing source of truth for configuration.
-- Source health and network metrics are available through `/api/health` and `/api/metrics`.
-- If Gitea Registry authentication is unavailable locally, build and push with the commands documented in `README.md`.
+Added or hardened:
+
+- `GET /api/health`
+- `GET /api/data`
+- `GET /api/metrics`
+- `POST /api/sweep`
+- `POST /api/action`
+
+Health now reports:
+
+- `starting`
+- `healthy`
+- `degraded`
+- `stale`
+- `error`
+
+It also reports:
+
+- last sweep timestamps
+- stale/bootstrap state
+- data age
+- source health
+- source errors
+- LLM configuration state
+- Telegram/Discord enabled state
+- memory store state
+
+### Live Data And Source Degradation
+
+- Existing `runs/latest.json` is only treated as bootstrap/stale data until a real sweep completes.
+- Sweeps update `sourceHealth`, SSE/API data, and memory state.
+- RSS/news feed failures no longer silently look like fresh valid data.
+- `safeFetch` now tracks request counts, failures, bytes, source labels, hosts, and recent fetch events.
+- `safeFetch` has better timeout/retry/backoff/error behavior and reports HTML-as-API-error cases.
+- Yahoo Finance fetches are more explicit about source errors and HTML/API failures.
+- ACLED missing credentials now degrade transparently.
+- Telegram polling has quieter network-error backoff logs.
+
+### LLM Integration
+
+Added unified OpenAI-compatible provider layer:
+
+```text
+lib/llm/openai-compatible.mjs
+```
+
+Supported provider paths include:
+
+- `openrouter`
+- `openai`
+- `openai-compatible`
+- `local-openai`
+- `lmstudio`
+- `lm-studio`
+- `ollama`
+
+Relevant environment keys:
+
+```text
+LLM_PROVIDER
+LLM_BASE_URL
+LLM_API_KEY
+LLM_MODEL
+LLM_TEMPERATURE
+LLM_MAX_TOKENS
+LLM_TIMEOUT_MS
+OPENROUTER_SITE_URL
+OPENROUTER_APP_NAME
+```
+
+OpenRouter Free and local OpenAI-compatible endpoints are documented in `README.md` and `.env.example`.
+
+### Memory
+
+Added Phase-1 SQLite memory:
+
+```text
+lib/intelligence-store.mjs
+runs/intelligence.db
+```
+
+It uses `node:sqlite` when available and gracefully falls back when unavailable.
+
+### Dashboard
+
+Implemented:
+
+- interactive Sensor Grid layer modes
+- focus/hide/normal states persisted in `localStorage`
+- Space Watch icon/orbit toggle
+- map/globe filtering consistency
+- flat map label redraw handling
+- live server-mode data loading from `/api/data` even when `jarvis.html` still contains an offline inline snapshot
+- Terminal Actions panel with `Status`, `Sweep`, and `Brief` buttons
+
+Important UI markers in the final code:
+
+```text
+layerModes
+spaceDisplayMode
+toggleSpaceDisplay()
+shouldShowType()
+runTerminalAction()
+```
+
+### Briefings
+
+Brief output now includes:
+
+- Source Integrity
+- evidence links
+- event IDs
+- configurable verbosity through `BRIEF_VERBOSITY`
+
+### Documentation
+
+Updated:
+
+- `README.md`
+- `.env.example`
+- `docs/sources/README.md`
+- `docs/sources/opensky.md`
+- `docs/sources/acled.md`
+- `docs/sources/telegram.md`
+- `docs/sources/firms.md`
+- `docs/sources/maritime.md`
+- `docs/security-review.md`
+- `docs/release-checklist.md`
+
+README includes:
+
+- Gitea Registry pull example
+- Dockge-compatible compose example
+- full `.env` examples
+- OpenRouter Free setup
+- LM Studio setup
+- Ollama setup
+- local OpenAI-compatible setup
+- Pangolin/reverse proxy notes
+
+## Registry And Images
+
+Registry image:
+
+```text
+git.wilkensxl.de/mrsphay/intelligence-terminal
+```
+
+Verified package tags through Gitea API:
+
+```text
+latest
+20260517
+e933586b220656a2858d2215b934b22d1f08a908
+53470cc701ec322080a89d220aef449b25850590
+```
+
+Successful pull test:
+
+```bash
+docker pull git.wilkensxl.de/mrsphay/intelligence-terminal:latest
+```
+
+Observed digest:
+
+```text
+sha256:780a41413921bd9a676461eca1cd1372591f523be4b7c9513d9bc085cbe7922d
+```
+
+## Gitea Actions
+
+Workflows present:
+
+```text
+.gitea/workflows/build.yml
+.gitea/workflows/security-scan.yml
+.gitea/workflows/repo-cleanup.yml
+.gitea/workflows/dependency-check.yml
+.gitea/workflows/release-dry-run.yml
+.gitea/workflows/template-compliance.yml
+```
+
+Final runs for commit `53470cc701ec322080a89d220aef449b25850590` were polled through the Gitea API and succeeded:
+
+```text
+build.yml on main: success
+build.yml on codex/production-intelligence-terminal: success
+release-dry-run.yml on main: success
+release-dry-run.yml on codex/production-intelligence-terminal: success
+template-compliance.yml on main: success
+template-compliance.yml on codex/production-intelligence-terminal: success
+```
+
+Relevant run URLs:
+
+```text
+https://git.wilkensxl.de/MrSphay/intelligence-terminal/actions/runs/23
+https://git.wilkensxl.de/MrSphay/intelligence-terminal/actions/runs/24
+https://git.wilkensxl.de/MrSphay/intelligence-terminal/actions/runs/25
+https://git.wilkensxl.de/MrSphay/intelligence-terminal/actions/runs/26
+https://git.wilkensxl.de/MrSphay/intelligence-terminal/actions/runs/27
+https://git.wilkensxl.de/MrSphay/intelligence-terminal/actions/runs/28
+```
+
+Repository secret expected by the registry publish workflow:
+
+```text
+REGISTRY_TOKEN
+```
+
+Local token note:
+
+- `GITEA_TOKEN` was visible in the final Codex process.
+- It was used only for Gitea API checks and not printed.
+
+## Issue Sync
+
+Open upstream GitHub issues were reviewed on 2026-05-17 from:
+
+```text
+https://github.com/calesthio/Crucix/issues
+```
+
+The upstream list contained 24 open issues. Issues already handled by this fork were not copied as open work, including the Docker stale-dashboard incident (#105), map label redraw (#70), Sensor Grid controls (#72), space display toggle (#51), source docs (#52), Dockge/CasaOS docs (#78), LLM timeout (#87), inject/static helper confusion (#100), network metrics (#101), Telegram polling backoff (#104), and briefing/evidence context (#75).
+
+Issues not relevant to this fork were also not copied, including the Wallpaper Engine redesign (#41), the fork-inflation discussion (#107), empty/unclear placeholders (#79/#80), and the general use-case discussion (#93).
+
+The following Gitea issues were created for real remaining work:
+
+```text
+#1 Reddit source must stop unauthenticated .json scraping
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/1
+
+#2 Send operator alerts when dashboard data remains stale
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/2
+
+#3 ACLED credentialed integration needs regression test and diagnostics
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/3
+
+#4 Complete memory and prediction loop beyond Phase-1 SQLite
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/4
+
+#5 Remove old inline dashboard snapshot from production builds
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/5
+
+#6 Harden Terminal Actions for public reverse-proxy deployments
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/6
+
+#7 Replace ADS-B stub with real disabled/degraded source handling
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/7
+
+#8 Clean inherited public-demo and upstream marketing references
+   https://git.wilkensxl.de/MrSphay/intelligence-terminal/issues/8
+```
+
+## Verification Already Performed
+
+Local lightweight checks:
+
+```bash
+npm run test:unit
+npm audit --omit=dev --audit-level=high
+docker compose --env-file .env.example config
+node --check server.mjs
+node --check dashboard/inject.mjs
+node --check lib/llm/openai-compatible.mjs
+git diff --check
+```
+
+Unit test result:
+
+```text
+21 tests passing
+0 failing
+```
+
+Audit result:
+
+```text
+0 high vulnerabilities
+```
+
+Docker build and smoke test were performed locally earlier:
+
+```bash
+docker build -t git.wilkensxl.de/mrsphay/intelligence-terminal:latest .
+docker run --rm -d --name intelligence-terminal-smoke -p 127.0.0.1::3117 -e AUTO_OPEN_BROWSER=false git.wilkensxl.de/mrsphay/intelligence-terminal:latest
+```
+
+Smoke test observations:
+
+- Server booted.
+- No `xdg-open` error.
+- Initial sweep completed.
+- `/api/health` moved from `starting` to `degraded` with transparent source errors.
+- Degraded state was expected without all optional API keys.
+
+Additional checks after fixing the dashboard live-data bug and Terminal Actions:
+
+```bash
+node --check server.mjs
+npm run test:unit
+docker compose --env-file .env.example config
+git diff --check
+```
+
+The dashboard script was also syntax-checked after extracting script blocks from `dashboard/public/jarvis.html`.
+
+## Important Commits
+
+```text
+7e85a54 chore: apply agent kit project structure
+85f97bb feat: harden intelligence runtime and llm providers
+42b7fc2 docs: add registry dockge and dashboard operations
+d072390 ci: align gitea workflows with agent kit
+0559481 ci: fix gitea registry publish login
+f3c9331 ci: fix agent kit compliance checks
+c2d572e fix: prepare runs volume before dropping privileges
+8e096b2 ci: harden gitea workflow reruns
+e933586 merge: reconcile main with production branch
+4262c7e docs: expand agent handoff
+53470cc fix: load live dashboard data and add terminal actions
+```
+
+The large implementation commit `85f97bb` and the dashboard/action fix `53470cc` are contained in both:
+
+```text
+origin/codex/production-intelligence-terminal
+origin/main
+```
+
+## How To Continue In A Fresh Codex Environment
+
+1. Clone the Gitea repository:
+
+```bash
+git clone https://git.wilkensxl.de/MrSphay/intelligence-terminal.git
+cd intelligence-terminal
+git checkout codex/production-intelligence-terminal
+```
+
+2. Confirm the expected commit:
+
+```bash
+git rev-parse HEAD
+```
+
+Expected:
+
+```text
+The branch tip should include commit 53470cc701ec322080a89d220aef449b25850590 and the later `docs: sync issue tracker and handoff` commit.
+```
+
+3. Read these files first:
+
+```text
+AGENTS.md
+.codex/project.md
+docs/agent-handoff.md
+README.md
+.env.example
+```
+
+4. If checking Actions, use `GITEA_TOKEN` from the environment. Do not print it.
+
+PowerShell check:
+
+```powershell
+if ($env:GITEA_TOKEN) { "GITEA_TOKEN=set" } else { "GITEA_TOKEN=missing" }
+```
+
+5. Useful commands:
+
+```bash
+npm run test:unit
+docker compose --env-file .env.example config
+docker pull git.wilkensxl.de/mrsphay/intelligence-terminal:latest
+```
+
+6. Start with Dockge/Pangolin using the README compose example and a `.env` based on `.env.example`.
+
+## Remaining Risks And Follow-Ups
+
+- Some sources will report `degraded` until optional keys are set, especially ACLED, FRED, EIA, and Cloudflare Radar.
+- OpenSky can rate-limit with HTTP 429; this is now visible in health instead of hidden.
+- GDELT/OFAC can time out under runner/network conditions; health reports this explicitly.
+- Browser-level visual verification of the full dashboard should be repeated after any future UI change.
+- The project still inherits the original Crucix broad source surface. Future work should prefer focused source-by-source tests over broad refactors.
+- If a new Codex environment sees non-fast-forward branch pushes, fetch first and preserve remote commits. Do not force-push without explicit approval.
+
+## Operator Pull Command
+
+For deployment:
+
+```bash
+docker pull git.wilkensxl.de/mrsphay/intelligence-terminal:latest
+```
+
+For a pinned deployment:
+
+```bash
+docker pull git.wilkensxl.de/mrsphay/intelligence-terminal:20260517
+```

docs/security-review.md

@@ -5,12 +5,21 @@
 - Shell execution: browser auto-open is gated by `AUTO_OPEN_BROWSER` and defaults to false.
 - Secrets: `.env` remains ignored; `.env.example` contains no real keys.
 - External network calls: source fetches use timeout/retry diagnostics and expose degraded state.
-- Manual actions: `/api/sweep` is local-only unless `SWEEP_TOKEN` is configured.
+- Manual actions: `/api/sweep` and `/api/action` are gated by `TERMINAL_ACTIONS_ENABLED` and local-only or `SWEEP_TOKEN` authorization.
 - File writes: runtime writes are limited to `runs/`.
 - HTML injection: dashboard data is JSON-injected only by the CLI path; server mode serves data through API/SSE.
 
+## Terminal Actions
+
+- `TERMINAL_ACTIONS_ENABLED=true` enables dashboard-triggered `status`, `sweep`, and `brief` actions through `POST /api/action`.
+- If `SWEEP_TOKEN` is set, callers must send the token through `x-sweep-token`, `Authorization: Bearer ...`, or the `token` request body field.
+- If `SWEEP_TOKEN` is empty, actions are accepted only from local loopback addresses.
+- For private Dockge/LAN deployments, this is intended to make the terminal operable from the browser.
+- For Pangolin or other internet-exposed deployments, set `SWEEP_TOKEN` or `TERMINAL_ACTIONS_ENABLED=false` until the public reverse-proxy hardening issue is completed.
+
 ## Residual Risk
 
 - External feeds can return malformed, stale, or adversarial content. UI rendering should continue to sanitize titles and URLs.
 - LLM outputs are advisory only and must not be treated as financial advice.
 - `node:sqlite` availability depends on the Node 22 build; when unavailable the memory database degrades to a no-op placeholder.
+- Browser-stored sweep tokens are acceptable for a trusted home-server UI, but should not be treated as a strong auth boundary on a public endpoint.

docs/sources/README.md

@@ -17,3 +17,4 @@ Source docs:
 - [FIRMS](firms.md)
 - [Maritime](maritime.md)
 - [ADS-B](adsb.md)
+- [Reddit](reddit.md)

docs/sources/acled.md

@@ -2,8 +2,11 @@
 
 Provides conflict events, fatalities, event types, and locations.
 
-- Auth: `ACLED_EMAIL` and `ACLED_PASSWORD`.
+- Auth: `ACLED_EMAIL` and `ACLED_PASSWORD`. `ACLED_USER` or `ACLED_USERNAME` may be used as aliases for `ACLED_EMAIL`.
 - Flow: OAuth password grant is tried first, then cookie session fallback.
-- Failure modes: missing credentials, terms/profile not completed, expired token, account missing API access.
-- Behavior: missing or rejected credentials produce degraded source health with the ACLED error text.
-- Test: set credentials, run `node apis/sources/acled.mjs`, then check `/api/health`.
+- Failure modes are classified as `no_credentials`, `auth_failed`, `access_denied`, or `api_failed`.
+- Behavior: missing, rejected, or unauthorized credentials produce degraded source health with a concise operator message.
+- Secret handling: debug output never prints bearer tokens, cookies, or the configured password.
+- Test: run `node --test test/acled-source.test.mjs`; with real credentials, run `node apis/sources/acled.mjs`, then check `/api/health`.
+
+`access_denied` normally means the login worked but the account cannot read API data. Check that ACLED terms are accepted, required profile fields are complete, and API access is enabled for the account.

docs/sources/opensky.md

@@ -6,4 +6,4 @@ Provides public aircraft state data for regional air-activity hotspots.
 - Failure modes: timeouts, `HTTP 429`, and empty regions.
 - Behavior: source health is marked degraded on API errors. The dashboard may use the most recent non-empty air snapshot from `runs/` and marks it in `airMeta.fallback`.
 - Test: start a sweep and inspect `/api/health` plus `airMeta` from `/api/data`.
-- Operator note: Crucix does not bypass OpenSky throttles. Increase `REFRESH_INTERVAL_MINUTES` if the source degrades repeatedly.
+- Operator note: Intelligence Terminal does not bypass OpenSky throttles. Increase `REFRESH_INTERVAL_MINUTES` if the source degrades repeatedly.

docs/sources/reddit.md

@@ -0,0 +1,33 @@
+# Reddit Source
+
+Reddit is used as a social sentiment input for selected geopolitical and market subreddits.
+
+## Configuration
+
+Create a Reddit script app at:
+
+```text
+https://www.reddit.com/prefs/apps/
+```
+
+Then set:
+
+```env
+REDDIT_CLIENT_ID=
+REDDIT_CLIENT_SECRET=
+```
+
+## Runtime Behavior
+
+- The source uses the OAuth client credentials flow and then reads `https://oauth.reddit.com`.
+- Unauthenticated `reddit.com/.../hot.json` scraping is intentionally disabled.
+- Missing credentials return `status: no_credentials` and are surfaced as source degradation.
+- OAuth failures return `status: auth_failed` without logging or returning the client secret.
+- Subreddit fetch failures return `status: degraded` with per-subreddit errors.
+
+## Test
+
+```bash
+node apis/sources/reddit.mjs
+npm run test:unit
+```