diff --git a/.env.example b/.env.example index 5d09163..ca66862 100644 --- a/.env.example +++ b/.env.example @@ -10,6 +10,9 @@ STALE_ALERT_COOLDOWN_MINUTES=60 DASHBOARD_URL= TERMINAL_ACTIONS_ENABLED=true SWEEP_TOKEN= +SSE_HEARTBEAT_INTERVAL_MS=25000 +TERMINAL_ACTION_RATE_LIMIT_WINDOW_MS=60000 +TERMINAL_ACTION_RATE_LIMIT_MAX=10 BRIEF_VERBOSITY=standard # LLM layer diff --git a/README.md b/README.md index 56363f8..a9bb0ff 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,9 @@ STALE_ALERT_COOLDOWN_MINUTES=60 DASHBOARD_URL=https://intelligence.example.internal TERMINAL_ACTIONS_ENABLED=true SWEEP_TOKEN= +SSE_HEARTBEAT_INTERVAL_MS=25000 +TERMINAL_ACTION_RATE_LIMIT_WINDOW_MS=60000 +TERMINAL_ACTION_RATE_LIMIT_MAX=10 BRIEF_VERBOSITY=standard LLM_PROVIDER=openrouter @@ -185,9 +188,66 @@ LLM_MODEL=your-model For Pangolin or another reverse proxy, forward HTTP traffic to `intelligence-terminal:3117` (or the `PORT` you set). Missing API keys do not crash sweeps; affected sources are reported as degraded in `/api/health`. +#### Terminal Action Exposure + +`POST /api/action` and `POST /api/sweep` can trigger operational actions such as manual sweeps. The dashboard has a **SET TOKEN** control that stores your `SWEEP_TOKEN` in browser local storage and sends it as the `x-crucix-token` header; do not put action tokens in URLs. + +Recommended settings: + +| Deployment | Settings | +| --- | --- | +| Private local machine | `NODE_ENV=development`, optional `SWEEP_TOKEN`, optional `TERMINAL_ACTIONS_ENABLED=true`. Localhost can run actions without a token for development. | +| Private LAN / Dockge | Set a strong `SWEEP_TOKEN`, keep `TERMINAL_ACTIONS_ENABLED=true`, expose only to trusted clients. | +| Pangolin-authenticated reverse proxy | Set a strong `SWEEP_TOKEN`, keep Pangolin auth in front, use the dashboard **SET TOKEN** flow once per browser. | +| Public internet | Do not expose Terminal Actions directly. If exposure is unavoidable, require `SWEEP_TOKEN`, keep proxy authentication enabled, lower `TERMINAL_ACTION_RATE_LIMIT_MAX`, and monitor server audit logs. | + +Action endpoints reject cross-origin POST origins, apply a small in-memory per-IP rate limit, and write sanitized audit lines without logging the token. + When data remains stale past `STALE_DATA_MAX_AGE_MINUTES`, the server sends an operator alert through configured Telegram/Discord channels after failed or degraded sweep attempts. `STALE_ALERT_COOLDOWN_MINUTES` prevents repeated stale alerts from spamming every refresh interval. Set `DASHBOARD_URL` to the Pangolin/public URL you want included in those alerts. -The dashboard Terminal Actions panel can trigger `status`, `sweep`, and `brief` through `/api/action`. Leave `TERMINAL_ACTIONS_ENABLED=true` for a private home-server deployment. For an internet-exposed deployment, set `SWEEP_TOKEN` and pass it through trusted automation, or set `TERMINAL_ACTIONS_ENABLED=false` to disable browser-triggered actions. If you protect actions with `SWEEP_TOKEN`, the browser can send it from `localStorage.crucix_sweep_token`. +#### Memory And Prediction Loop + +Crucix stores longitudinal memory in `runs/intelligence.db` when the current Node.js build exposes `node:sqlite`. If SQLite is unavailable, the file is created as a harmless placeholder and `/api/health` reports the memory store as unavailable instead of failing the sweep. + +The memory layer persists: + +| Table | Purpose | +| --- | --- | +| `runs` | Sweep timestamps, source health counts, and delta direction summaries. | +| `entities` | Stable entity IDs for recurring countries, regions, and locations. | +| `events` | Stable event IDs for conflict, OSINT, urgent news, and new delta signals across sweeps. | +| `predictions` | Trade/intelligence hypotheses with evidence, confidence, horizon, outcome state, and latest grading. | + +Query endpoints: + +```text +GET /api/memory/search?q=iran&limit=25 +GET /api/memory/predictions?state=open&limit=25 +``` + +Memory endpoints use the same operator authorization gate as Terminal Actions. The dashboard Terminal Actions panel includes a `Memory` action for a quick operator-facing view of recent events and prediction states. + +Retention, backup, and privacy expectations: + +- Treat `runs/intelligence.db` as operator data. It can contain source excerpts, headlines, generated hypotheses, and URLs from your configured feeds. +- Back up `runs/` with the rest of your Dockge volume if you want longitudinal learning to survive container replacement. +- Delete `runs/intelligence.db` to reset SQLite memory; the next sweep recreates the schema. +- Do not commit `runs/` or `.env`. API credentials stay in `.env`; memory stores derived observations, not secrets. +- If you expose the dashboard through a reverse proxy, protect Terminal Actions and memory queries behind your normal authentication boundary. + +#### Reverse Proxy SSE + +The dashboard receives live sweep updates from `GET /events` using Server-Sent Events. The server sends `retry: 10000` reconnect guidance and lightweight heartbeat comments every `SSE_HEARTBEAT_INTERVAL_MS` milliseconds so reverse proxies do not close an otherwise idle stream between 15-minute sweeps. + +Recommended proxy settings: + +| Proxy | Setting | +| --- | --- | +| Pangolin / Traefik-style frontends | Keep response streaming enabled and set idle timeouts above `SSE_HEARTBEAT_INTERVAL_MS`. | +| Nginx | Disable proxy buffering for `/events`, keep `proxy_read_timeout` above the heartbeat interval, and preserve `Connection: keep-alive`. | +| Cloudflare-style proxies | Keep the heartbeat below common idle cutoffs; the default 25s is intentionally conservative. | + +If you raise the heartbeat interval, keep it shorter than the lowest idle timeout in the proxy chain. `/api/metrics` includes network health grouped by host and source/provider. Source modules should use `safeFetch(url, { source: 'SourceName' })`; when omitted, the shared helper infers a stable provider bucket from the URL host instead of grouping normal source traffic under `unknown`. Raw fetch exceptions are documented in [Source Fetch Instrumentation](docs/source-fetch-instrumentation.md). diff --git a/crucix.config.mjs b/crucix.config.mjs index c0604b0..b056dfa 100644 --- a/crucix.config.mjs +++ b/crucix.config.mjs @@ -26,7 +26,10 @@ export default { staleAlertCooldownMinutes: intEnv('STALE_ALERT_COOLDOWN_MINUTES', 60), dashboardUrl: process.env.DASHBOARD_URL || null, sweepToken: process.env.SWEEP_TOKEN || null, - terminalActionsEnabled: boolEnv('TERMINAL_ACTIONS_ENABLED', true), + terminalActionsEnabled: boolEnv('TERMINAL_ACTIONS_ENABLED', !!process.env.SWEEP_TOKEN || process.env.NODE_ENV !== 'production'), + terminalActionRateLimitWindowMs: intEnv('TERMINAL_ACTION_RATE_LIMIT_WINDOW_MS', 60_000), + terminalActionRateLimitMax: intEnv('TERMINAL_ACTION_RATE_LIMIT_MAX', 10), + sseHeartbeatIntervalMs: intEnv('SSE_HEARTBEAT_INTERVAL_MS', 25000), llm: { provider: process.env.LLM_PROVIDER || null, // anthropic | openai | gemini | codex | openrouter | minimax | mistral | ollama | grok diff --git a/dashboard/public/jarvis.html b/dashboard/public/jarvis.html index 8c14f2a..cb0b228 100644 --- a/dashboard/public/jarvis.html +++ b/dashboard/public/jarvis.html @@ -83,7 +83,7 @@ html,body{height:100%;background:var(--bg);color:var(--text);font-family:var(--s .sensor-actions{display:flex;gap:6px;align-items:center} .mini-btn{border:1px solid rgba(100,240,200,0.18);background:rgba(100,240,200,0.04);color:var(--dim);font-family:var(--mono);font-size:9px;padding:3px 6px;cursor:pointer} .mini-btn:hover{color:var(--accent);border-color:rgba(100,240,200,0.4)} -.action-grid{display:grid;grid-template-columns:repeat(3,1fr);gap:6px;margin-bottom:10px} +.action-grid{display:grid;grid-template-columns:repeat(4,1fr);gap:6px;margin-bottom:10px} .action-btn{border:1px solid rgba(68,204,255,0.24);background:rgba(68,204,255,0.06);color:var(--text);font-family:var(--mono);font-size:9px;padding:7px 6px;cursor:pointer;text-transform:uppercase;letter-spacing:.08em} .action-btn:hover{border-color:rgba(68,204,255,0.55);color:var(--accent2);background:rgba(68,204,255,0.12)} .action-btn[disabled]{opacity:.45;cursor:wait} @@ -432,6 +432,7 @@ let terminalOutput = 'Ready. Live data is loaded from /api/data in server mode.' let terminalBusy = false; let currentRegion = 'world'; let flatSvg, flatProjection, flatPath, flatG, flatZoom, flatW, flatH; +const terminalActionTokenKey = 'crucix_sweep_token'; const layerTypeMap = { air: ['air'], @@ -632,6 +633,7 @@ function renderTopbar(){ const ts = new Date(D.meta.timestamp); const d = ts.toLocaleDateString('en-US',{month:'short',day:'numeric',year:'numeric'}).toUpperCase(); const timeStr = ts.toLocaleTimeString('en-US',{hour:'2-digit',minute:'2-digit',hour12:true}); + const hasActionToken = !!getTerminalActionToken(); document.getElementById('topbar').innerHTML=`
CRUCIX MONITOR @@ -644,12 +646,26 @@ function renderTopbar(){ ${d} ${timeStr} ${t('dashboard.sources','SOURCES')} ${D.meta.sourcesOk}/${D.meta.sourcesQueried} ${D.delta?.summary ? `${t('dashboard.delta','DELTA')} ${D.delta.summary.direction==='risk-off'?'▲ '+t('dashboard.riskOff','RISK-OFF'):D.delta.summary.direction==='risk-on'?'▼ '+t('dashboard.riskOn','RISK-ON'):'◆ '+t('dashboard.mixed','MIXED')}` : ''} + ${t('dashboard.highAlert','HIGH ALERT')}
`; renderRegionControls(); } +function getTerminalActionToken(){ + return localStorage.getItem(terminalActionTokenKey) || localStorage.getItem('crucix_terminal_action_token') || ''; +} + +function configureTerminalActionToken(){ + const next = window.prompt('Terminal action token (SWEEP_TOKEN). Leave empty to clear.', getTerminalActionToken()); + if(next === null) return; + const clean = next.trim(); + if(clean) localStorage.setItem(terminalActionTokenKey, clean); + else localStorage.removeItem(terminalActionTokenKey); + renderTopbar(); +} + // === LEFT RAIL === function layerMode(key){ return layerModes[key] || 'normal'; } function layerModeLabel(key){ return layerMode(key) === 'focus' ? 'focused' : layerMode(key) === 'hidden' ? 'hidden' : 'normal'; } @@ -1592,6 +1608,12 @@ function renderLower(){ async function runTerminalAction(action){ if(terminalBusy) return; + let token = getTerminalActionToken(); + if(!token && location.hostname !== 'localhost' && location.hostname !== '127.0.0.1'){ + configureTerminalActionToken(); + token = getTerminalActionToken(); + if(!token) return; + } terminalBusy = true; terminalOutput = `> ${action}\nRunning...`; renderRight(); @@ -1600,7 +1622,7 @@ async function runTerminalAction(action){ method:'POST', headers:{ 'Content-Type':'application/json', - ...(localStorage.getItem('crucix_sweep_token') ? {'x-crucix-token': localStorage.getItem('crucix_sweep_token')} : {}) + ...(token ? {'x-crucix-token': token} : {}) }, body:JSON.stringify({action}) }); @@ -1619,6 +1641,17 @@ async function runTerminalAction(action){ ].join('\n'); }else if(action === 'brief'){ terminalOutput = `> brief\n${payload.text || 'No briefing text returned.'}`; + }else if(action === 'memory'){ + const events = payload.recentEvents || []; + const predictions = payload.predictions || []; + terminalOutput = [ + '> memory', + `Store: ${payload.memory?.available ? 'available' : 'unavailable'}`, + `Recent events: ${events.length}`, + ...events.slice(0,4).map(e => `- ${e.kind}: ${e.name}${e.region ? ' [' + e.region + ']' : ''}`), + `Predictions: ${predictions.length}`, + ...predictions.slice(0,4).map(p => `- ${p.outcome_state || 'open'}: ${p.title}`) + ].join('\n'); }else if(action === 'sweep'){ terminalOutput = `> sweep\n${payload.status === 'already_running' ? 'Sweep already running.' : 'Sweep accepted. The dashboard will update when the sweep finishes.'}`; } @@ -1685,7 +1718,9 @@ function renderRight(){ + +
${terminalOutput.replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])).replace(/\n/g,'
')}
diff --git a/lib/intelligence-store.mjs b/lib/intelligence-store.mjs index 10daf40..fff20d3 100644 --- a/lib/intelligence-store.mjs +++ b/lib/intelligence-store.mjs @@ -2,6 +2,9 @@ import { existsSync, mkdirSync, writeFileSync } from 'fs'; import { dirname, join } from 'path'; +import { createHash } from 'crypto'; + +const PREDICTION_STATES = new Set(['open', 'monitoring', 'observed', 'expired_unverified', 'invalidated']); export class IntelligenceStore { constructor(dbPath) { @@ -30,15 +33,24 @@ export class IntelligenceStore { ); CREATE TABLE IF NOT EXISTS predictions ( id INTEGER PRIMARY KEY AUTOINCREMENT, + stable_id TEXT UNIQUE, created_at TEXT NOT NULL, + updated_at TEXT, title TEXT NOT NULL, type TEXT, + hypothesis TEXT, + evidence_json TEXT, confidence TEXT, + horizon TEXT, + outcome_state TEXT DEFAULT 'open', + outcome_json TEXT, + last_evaluated_at TEXT, source TEXT, payload_json TEXT NOT NULL ); CREATE TABLE IF NOT EXISTS entities ( id INTEGER PRIMARY KEY AUTOINCREMENT, + stable_id TEXT UNIQUE, first_seen TEXT NOT NULL, last_seen TEXT NOT NULL, name TEXT NOT NULL, @@ -46,7 +58,21 @@ export class IntelligenceStore { count INTEGER DEFAULT 1, UNIQUE(name, kind) ); + CREATE TABLE IF NOT EXISTS events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + stable_id TEXT NOT NULL UNIQUE, + first_seen TEXT NOT NULL, + last_seen TEXT NOT NULL, + kind TEXT NOT NULL, + name TEXT NOT NULL, + region TEXT, + severity TEXT, + source TEXT, + evidence_json TEXT NOT NULL, + count INTEGER DEFAULT 1 + ); `); + this._migrate(); this.available = true; } catch (err) { this.available = false; @@ -71,24 +97,141 @@ export class IntelligenceStore { delta?.summary?.direction || null, JSON.stringify({ meta, delta: delta?.summary || null }), ); - for (const idea of data.ideas || []) { - this.db.prepare(`INSERT INTO predictions (created_at, title, type, confidence, source, payload_json) - VALUES (?, ?, ?, ?, ?, ?)`).run( - timestamp, - idea.title || 'Untitled idea', - idea.type || null, - idea.confidence || null, - idea.source || data.ideasSource || null, - JSON.stringify(idea), - ); - } this._recordEntities(data, timestamp); + this._recordEvents(data, delta, timestamp); + this.evaluatePredictions(data, timestamp); + this._recordPredictions(data, timestamp); } status() { return { available: this.available, path: this.dbPath, reason: this.reason }; } + queryMemory({ q = '', limit = 25 } = {}) { + if (!this.available || !this.db) return { available: false, reason: this.reason, results: [] }; + const safeLimit = Math.max(1, Math.min(100, Number(limit) || 25)); + const term = String(q || '').trim(); + const like = `%${term}%`; + const where = term + ? 'WHERE name LIKE ? OR region LIKE ? OR source LIKE ? OR kind LIKE ?' + : ''; + const params = term ? [like, like, like, like, safeLimit] : [safeLimit]; + const events = this.db.prepare(` + SELECT stable_id, first_seen, last_seen, kind, name, region, severity, source, count, evidence_json + FROM events + ${where} + ORDER BY last_seen DESC + LIMIT ? + `).all(...params).map(row => ({ ...row, evidence: parseJson(row.evidence_json, {}) })); + return { available: true, q: term, results: events }; + } + + listPredictions({ state = null, limit = 25 } = {}) { + if (!this.available || !this.db) return { available: false, reason: this.reason, predictions: [] }; + const safeLimit = Math.max(1, Math.min(100, Number(limit) || 25)); + const normalizedState = state && PREDICTION_STATES.has(String(state)) ? String(state) : null; + const rows = normalizedState + ? this.db.prepare(`SELECT * FROM predictions WHERE outcome_state = ? ORDER BY created_at DESC LIMIT ?`).all(normalizedState, safeLimit) + : this.db.prepare(`SELECT * FROM predictions ORDER BY created_at DESC LIMIT ?`).all(safeLimit); + return { + available: true, + predictions: rows.map(row => ({ + stable_id: row.stable_id, + created_at: row.created_at, + updated_at: row.updated_at, + title: row.title, + type: row.type, + hypothesis: row.hypothesis, + confidence: row.confidence, + horizon: row.horizon, + outcome_state: row.outcome_state, + last_evaluated_at: row.last_evaluated_at, + source: row.source, + evidence: parseJson(row.evidence_json, []), + outcome: parseJson(row.outcome_json, null), + })), + }; + } + + evaluatePredictions(data, timestamp = new Date().toISOString()) { + if (!this.available || !this.db) return; + const rows = this.db.prepare(` + SELECT id, created_at, title, type, horizon, outcome_state, payload_json + FROM predictions + WHERE outcome_state IN ('open', 'monitoring') + ORDER BY created_at ASC + LIMIT 200 + `).all(); + for (const row of rows) { + const payload = parseJson(row.payload_json, {}); + const evaluation = evaluatePredictionAgainstSweep(row, payload, data, timestamp); + this.db.prepare(`UPDATE predictions + SET outcome_state = ?, outcome_json = ?, last_evaluated_at = ?, updated_at = ? + WHERE id = ?`).run( + evaluation.state, + JSON.stringify(evaluation), + timestamp, + timestamp, + row.id, + ); + } + } + + _migrate() { + const columns = { + predictions: [ + ['stable_id', 'TEXT'], + ['updated_at', 'TEXT'], + ['hypothesis', 'TEXT'], + ['evidence_json', 'TEXT'], + ['horizon', 'TEXT'], + ['outcome_state', "TEXT DEFAULT 'open'"], + ['outcome_json', 'TEXT'], + ['last_evaluated_at', 'TEXT'], + ], + entities: [ + ['stable_id', 'TEXT'], + ], + }; + for (const [table, defs] of Object.entries(columns)) { + for (const [name, type] of defs) { + try { this.db.exec(`ALTER TABLE ${table} ADD COLUMN ${name} ${type}`); } catch { } + } + } + try { this.db.exec(`CREATE UNIQUE INDEX IF NOT EXISTS idx_predictions_stable_id ON predictions(stable_id)`); } catch { } + try { this.db.exec(`CREATE UNIQUE INDEX IF NOT EXISTS idx_entities_stable_id ON entities(stable_id)`); } catch { } + } + + _recordPredictions(data, timestamp) { + for (const idea of data.ideas || []) { + const title = idea.title || 'Untitled idea'; + const stableId = stableId('prediction', title, idea.type || '', idea.ticker || '', idea.horizon || ''); + const evidence = Array.isArray(idea.signals) ? idea.signals : []; + this.db.prepare(`INSERT INTO predictions ( + stable_id, created_at, updated_at, title, type, hypothesis, evidence_json, confidence, + horizon, outcome_state, source, payload_json + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'open', ?, ?) + ON CONFLICT(stable_id) DO UPDATE SET + updated_at=excluded.updated_at, + confidence=excluded.confidence, + evidence_json=excluded.evidence_json, + payload_json=excluded.payload_json`).run( + stableId, + timestamp, + timestamp, + title, + idea.type || null, + idea.rationale || idea.text || title, + JSON.stringify(evidence), + idea.confidence || null, + idea.horizon || null, + idea.source || data.ideasSource || null, + JSON.stringify(idea), + ); + } + } + _recordEntities(data, timestamp) { const names = []; for (const item of data.acled?.deadliestEvents || []) { @@ -99,14 +242,154 @@ export class IntelligenceStore { if (item.region) names.push([item.region, 'region']); } for (const [name, kind] of names.slice(0, 200)) { - this.db.prepare(`INSERT INTO entities (first_seen, last_seen, name, kind, count) - VALUES (?, ?, ?, ?, 1) + const cleanName = String(name).slice(0, 160); + this.db.prepare(`INSERT INTO entities (stable_id, first_seen, last_seen, name, kind, count) + VALUES (?, ?, ?, ?, ?, 1) ON CONFLICT(name, kind) DO UPDATE SET last_seen=excluded.last_seen, count=count+1`).run( + stableId('entity', kind, cleanName), timestamp, timestamp, - String(name).slice(0, 160), + cleanName, kind, ); } } + + _recordEvents(data, delta, timestamp) { + const events = extractEvents(data, delta); + for (const event of events.slice(0, 300)) { + this.db.prepare(`INSERT INTO events ( + stable_id, first_seen, last_seen, kind, name, region, severity, source, evidence_json, count + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 1) + ON CONFLICT(stable_id) DO UPDATE SET + last_seen=excluded.last_seen, + severity=COALESCE(excluded.severity, severity), + evidence_json=excluded.evidence_json, + count=count+1`).run( + event.stable_id, + timestamp, + timestamp, + event.kind, + event.name, + event.region || null, + event.severity || null, + event.source || null, + JSON.stringify(event.evidence || {}), + ); + } + } +} + +function stableId(...parts) { + const input = parts.map(part => String(part || '').trim().toLowerCase()).join('|'); + return createHash('sha256').update(input).digest('hex').slice(0, 24); +} + +function parseJson(value, fallback) { + try { return value ? JSON.parse(value) : fallback; } catch { return fallback; } +} + +function extractEvents(data, delta) { + const events = []; + const push = ({ kind, name, region, severity, source, evidence }) => { + if (!kind || !name) return; + events.push({ + stable_id: stableId('event', kind, name, region || source || ''), + kind, + name: String(name).slice(0, 240), + region: region ? String(region).slice(0, 120) : null, + severity: severity || null, + source: source || null, + evidence: evidence || {}, + }); + }; + + for (const item of data.acled?.deadliestEvents || []) { + push({ + kind: 'conflict', + name: item.event_type || item.sub_event_type || item.location || item.country, + region: item.country || item.location, + severity: Number(item.fatalities || 0) > 0 ? 'high' : 'medium', + source: 'ACLED', + evidence: item, + }); + } + for (const item of data.tg?.urgent || []) { + push({ + kind: 'osint', + name: (item.text || '').slice(0, 120), + region: item.region || 'OSINT', + severity: 'high', + source: item.channel || item.chat || 'telegram', + evidence: item, + }); + } + for (const item of data.newsFeed || data.news || []) { + if (!item.urgent) continue; + push({ + kind: 'news', + name: item.headline || item.title, + region: item.region, + severity: 'medium', + source: item.source, + evidence: item, + }); + } + for (const signal of delta?.signals?.new || []) { + push({ + kind: 'delta', + name: signal.label || signal.reason || signal.key, + region: signal.region, + severity: signal.severity || 'medium', + source: 'delta', + evidence: signal, + }); + } + return events; +} + +function evaluatePredictionAgainstSweep(row, payload, data, timestamp) { + const terms = [ + row.title, + payload.ticker, + ...(Array.isArray(payload.signals) ? payload.signals : []), + ].filter(Boolean).map(v => String(v).toLowerCase()); + const evidenceText = [ + ...(data.tSignals || []), + ...(data.newsFeed || []).slice(0, 40).map(n => `${n.source || ''} ${n.headline || n.title || ''}`), + ...(data.tg?.urgent || []).slice(0, 20).map(p => p.text || ''), + ].join('\n').toLowerCase(); + const matched = terms.filter(term => term.length >= 4 && evidenceText.includes(term.slice(0, 60))); + const expired = predictionExpired(row.created_at, row.horizon, timestamp); + const state = matched.length + ? 'observed' + : expired + ? 'expired_unverified' + : 'monitoring'; + return { + state, + evaluated_at: timestamp, + matched_terms: matched.slice(0, 10), + expired, + reason: matched.length + ? 'Current sweep contains matching evidence terms.' + : expired + ? 'Prediction horizon elapsed without matching evidence.' + : 'Prediction remains open for future sweeps.', + }; +} + +function predictionExpired(createdAt, horizon, nowIso) { + const created = new Date(createdAt).getTime(); + const now = new Date(nowIso).getTime(); + if (!Number.isFinite(created) || !Number.isFinite(now)) return false; + const text = String(horizon || '').toLowerCase(); + const days = text.includes('intraday') ? 1 + : text.includes('day') ? 7 + : text.includes('week') ? 45 + : text.includes('month') ? 180 + : text.includes('strategic') ? 365 + : 30; + return now - created > days * 24 * 60 * 60 * 1000; } diff --git a/server.mjs b/server.mjs index eb661c6..b2c38ce 100644 --- a/server.mjs +++ b/server.mjs @@ -41,6 +41,7 @@ let sweepStartedAt = null; // Timestamp when current/last sweep started let sweepInProgress = false; const startTime = Date.now(); const sseClients = new Set(); +const terminalActionBuckets = new Map(); const staleAlertState = {}; // === Delta/Memory === @@ -291,29 +292,67 @@ app.get('/api/metrics', (req, res) => { }); }); -app.post('/api/sweep', express.json(), (req, res) => { - if (!canRunTerminalAction(req)) return res.status(403).json({ error: 'Terminal actions disabled or unauthorized' }); - triggerSweep(res); +app.get('/api/memory/search', (req, res) => { + const guard = authorizeTerminalAction(req, res, 'memory:search'); + if (!guard.ok) return; + auditTerminalAction(req, 'memory:search', 'ok'); + res.json(intelligenceStore.queryMemory({ + q: req.query.q || '', + limit: req.query.limit || 25, + })); }); -app.post('/api/action', express.json(), async (req, res) => { - if (!canRunTerminalAction(req)) return res.status(403).json({ error: 'Terminal actions disabled or unauthorized' }); - const action = String(req.body?.action || req.query.action || '').toLowerCase(); +app.get('/api/memory/predictions', (req, res) => { + const guard = authorizeTerminalAction(req, res, 'memory:predictions'); + if (!guard.ok) return; + auditTerminalAction(req, 'memory:predictions', 'ok'); + res.json(intelligenceStore.listPredictions({ + state: req.query.state || null, + limit: req.query.limit || 25, + })); +}); + +app.post('/api/sweep', express.json(), (req, res) => { + const guard = authorizeTerminalAction(req, res, 'sweep'); + if (!guard.ok) return; + triggerSweepAction(req, res, 'sweep'); +}); + +app.post('/api/action', express.json(), (req, res) => { + const action = String(req.body?.action || req.body?.command || '').trim().toLowerCase(); + const guard = authorizeTerminalAction(req, res, action || 'unknown'); + if (!guard.ok) return; if (action === 'status') { - return res.json({ ok: true, action, health: buildHealth() }); + auditTerminalAction(req, 'status', 'ok'); + return res.json({ ok: true, action, status: 'ok', health: buildHealth() }); } if (action === 'brief') { - if (!currentData) return res.status(503).json({ ok: false, action, error: 'No data yet — first sweep in progress' }); - return res.json({ ok: true, action, text: buildBrief(currentData) }); + if (!currentData) { + auditTerminalAction(req, 'brief', 'rejected', 'no_data'); + return res.status(503).json({ ok: false, action, error: 'No data yet - first sweep in progress' }); + } + auditTerminalAction(req, 'brief', 'ok'); + const brief = buildBrief(currentData); + return res.json({ ok: true, action, status: 'ok', brief, text: brief }); } - if (action === 'sweep') { - return triggerSweep(res); + if (action === 'memory') { + auditTerminalAction(req, 'memory', 'ok'); + return res.json({ + ok: true, + action, + memory: intelligenceStore.status(), + recentEvents: intelligenceStore.queryMemory({ q: req.body?.q || '', limit: 8 }).results, + predictions: intelligenceStore.listPredictions({ limit: 8 }).predictions, + }); } - res.status(400).json({ ok: false, error: 'Unknown action', actions: ['status', 'brief', 'sweep'] }); + if (action === 'sweep') return triggerSweepAction(req, res, 'action:sweep'); + + auditTerminalAction(req, action || 'unknown', 'rejected', 'unknown_action'); + return res.status(400).json({ ok: false, error: 'Unknown action', allowed: ['status', 'brief', 'memory', 'sweep'], actions: ['status', 'brief', 'memory', 'sweep'] }); }); // API: available locales @@ -331,10 +370,24 @@ app.get('/events', (req, res) => { 'Cache-Control': 'no-cache', 'Connection': 'keep-alive', 'Access-Control-Allow-Origin': '*', + 'X-Accel-Buffering': 'no', }); + res.write('retry: 10000\n'); res.write('data: {"type":"connected"}\n\n'); + const heartbeatMs = Math.max(5000, config.sseHeartbeatIntervalMs || 25000); + const heartbeat = setInterval(() => { + try { + res.write(`: heartbeat ${new Date().toISOString()}\n\n`); + } catch { + clearInterval(heartbeat); + sseClients.delete(res); + } + }, heartbeatMs); sseClients.add(res); - req.on('close', () => sseClients.delete(res)); + req.on('close', () => { + clearInterval(heartbeat); + sseClients.delete(res); + }); }); function broadcast(data) { @@ -344,26 +397,114 @@ function broadcast(data) { } } +function requestIp(req) { + return req.ip || req.socket?.remoteAddress || 'unknown'; +} + +function isLocalRequest(req) { + const remote = requestIp(req); + return remote === '::1' + || remote === '127.0.0.1' + || remote === '::ffff:127.0.0.1' + || remote.startsWith('127.') + || remote === 'localhost'; +} + +function sameOriginPost(req) { + const origin = req.get('origin'); + if (!origin) return true; + try { + const originUrl = new URL(origin); + const host = req.get('host'); + return host && originUrl.host === host; + } catch { + return false; + } +} + +function actionToken(req) { + return req.get('x-crucix-token') || req.body?.token || null; +} + +function auditTerminalAction(req, action, outcome, detail = null) { + const suffix = detail ? ` detail=${detail}` : ''; + console.log(`[Crucix][audit] terminal_action action=${action || 'unknown'} outcome=${outcome} ip=${requestIp(req)}${suffix}`); +} + +function rateLimitTerminalAction(req, action) { + const now = Date.now(); + const windowMs = Math.max(1000, config.terminalActionRateLimitWindowMs || 60_000); + const max = Math.max(1, config.terminalActionRateLimitMax || 10); + const key = `${requestIp(req)}:${action}`; + const bucket = terminalActionBuckets.get(key); + if (!bucket || now > bucket.resetAt) { + terminalActionBuckets.set(key, { count: 1, resetAt: now + windowMs }); + return { ok: true }; + } + bucket.count += 1; + if (bucket.count > max) { + return { ok: false, retryAfterSeconds: Math.ceil((bucket.resetAt - now) / 1000) }; + } + return { ok: true }; +} + +function authorizeTerminalAction(req, res, action) { + const rate = rateLimitTerminalAction(req, action); + if (!rate.ok) { + auditTerminalAction(req, action, 'rejected', 'rate_limited'); + res.set('Retry-After', String(rate.retryAfterSeconds)); + res.status(429).json({ error: 'Too many terminal actions', retryAfterSeconds: rate.retryAfterSeconds }); + return { ok: false }; + } + + if (!sameOriginPost(req)) { + auditTerminalAction(req, action, 'rejected', 'csrf_origin'); + res.status(403).json({ error: 'Origin mismatch' }); + return { ok: false }; + } + + const local = isLocalRequest(req); + const token = actionToken(req); + if (!config.terminalActionsEnabled) { + auditTerminalAction(req, action, 'rejected', 'disabled'); + res.status(403).json({ error: 'Terminal actions are disabled' }); + return { ok: false }; + } + + if (config.sweepToken) { + if (token !== config.sweepToken) { + auditTerminalAction(req, action, 'rejected', 'invalid_token'); + res.status(401).json({ error: 'Invalid terminal action token' }); + return { ok: false }; + } + return { ok: true }; + } + + if (!local) { + auditTerminalAction(req, action, 'rejected', 'missing_token'); + res.status(403).json({ error: 'Terminal actions are local-only unless SWEEP_TOKEN is set' }); + return { ok: false }; + } + + return { ok: true }; +} + +function triggerSweepAction(req, res, auditAction) { + if (sweepInProgress) { + auditTerminalAction(req, auditAction, 'rejected', 'already_running'); + return res.status(409).json({ ok: true, status: 'already_running', sweepStartedAt }); + } + auditTerminalAction(req, auditAction, 'accepted'); + runSweepCycle().catch(err => console.error('[Crucix] API-triggered sweep failed:', err.message)); + return res.status(202).json({ ok: true, status: 'accepted' }); +} + function dataAgeMs() { const ts = currentData?.meta?.timestamp || lastSuccessfulSweepTime || lastSweepTime; const ms = ts ? Date.now() - new Date(ts).getTime() : null; return Number.isFinite(ms) ? ms : null; } -function canRunTerminalAction(req) { - const remote = req.ip || ''; - const local = remote.includes('127.0.0.1') || remote === '::1' || remote === '::ffff:127.0.0.1'; - const token = req.get('x-crucix-token') || req.query.token || req.body?.token; - if (config.sweepToken) return token === config.sweepToken; - return Boolean(config.terminalActionsEnabled || local); -} - -function triggerSweep(res) { - if (sweepInProgress) return res.status(409).json({ ok: true, status: 'already_running', sweepStartedAt }); - runSweepCycle().catch(err => console.error('[Crucix] API-triggered sweep failed:', err.message)); - return res.status(202).json({ ok: true, status: 'accepted' }); -} - function getLLMStatus() { if (!config.llm.provider) return { state: 'disabled' }; if (!llmProvider) return { state: 'misconfigured', provider: config.llm.provider }; @@ -407,7 +548,8 @@ function buildHealth() { llm: getLLMStatus(), telegramEnabled: !!(config.telegram.botToken && config.telegram.chatId), discordEnabled: !!(config.discord?.botToken || config.discord?.webhookUrl), - terminalActionsEnabled: Boolean(config.terminalActionsEnabled || config.sweepToken), + terminalActionsEnabled: config.terminalActionsEnabled, + terminalActionsTokenRequired: !!config.sweepToken, refreshIntervalMinutes: config.refreshIntervalMinutes, language: currentLanguage, memory: intelligenceStore.status(), diff --git a/test/fetch-utils.test.mjs b/test/fetch-utils.test.mjs index 77d2a9c..a469796 100644 --- a/test/fetch-utils.test.mjs +++ b/test/fetch-utils.test.mjs @@ -126,6 +126,57 @@ test('inferFetchSource returns provider names and host fallback', () => { assert.equal(inferFetchSource('https://unknown.example.test/path'), 'unknown.example.test'); }); +test('SSE endpoint sends reconnect guidance and clears heartbeat timer', () => { + const server = readFileSync(new URL('../server.mjs', import.meta.url), 'utf8'); + const config = readFileSync(new URL('../crucix.config.mjs', import.meta.url), 'utf8'); + assert.match(config, /sseHeartbeatIntervalMs/); + assert.match(server, /retry: 10000\\n/); + assert.match(server, /setInterval\(\(\) =>/); + assert.match(server, /: heartbeat/); + assert.match(server, /clearInterval\(heartbeat\)/); + assert.match(server, /X-Accel-Buffering/); +}); + +test('intelligence store defines durable memory and prediction lifecycle tables', () => { + const store = readFileSync(new URL('../lib/intelligence-store.mjs', import.meta.url), 'utf8'); + assert.match(store, /CREATE TABLE IF NOT EXISTS events/); + assert.match(store, /stable_id TEXT NOT NULL UNIQUE/); + assert.match(store, /hypothesis TEXT/); + assert.match(store, /evidence_json TEXT/); + assert.match(store, /outcome_state TEXT DEFAULT 'open'/); + assert.match(store, /evaluatePredictions/); + assert.match(store, /queryMemory/); + assert.match(store, /listPredictions/); +}); + +test('server exposes memory-backed query APIs and dashboard memory action', () => { + const server = readFileSync(new URL('../server.mjs', import.meta.url), 'utf8'); + const html = readFileSync(new URL('../dashboard/public/jarvis.html', import.meta.url), 'utf8'); + assert.match(server, /\/api\/memory\/search/); + assert.match(server, /\/api\/memory\/predictions/); + assert.match(server, /action === 'memory'/); + assert.match(html, /runTerminalAction\('memory'\)/); +}); + +test('terminal action endpoints avoid URL tokens and include hardening gates', () => { + const server = readFileSync(new URL('../server.mjs', import.meta.url), 'utf8'); + assert.match(server, /app\.post\('\/api\/action'/); + assert.match(server, /app\.post\('\/api\/sweep'/); + assert.match(server, /x-crucix-token/); + assert.match(server, /sameOriginPost/); + assert.match(server, /rateLimitTerminalAction/); + assert.match(server, /auditTerminalAction/); + assert.doesNotMatch(server, /req\.query\.token/); +}); + +test('dashboard exposes token configuration flow without devtools edits', () => { + const html = readFileSync(new URL('../dashboard/public/jarvis.html', import.meta.url), 'utf8'); + assert.match(html, /configureTerminalActionToken/); + assert.match(html, /crucix_sweep_token/); + assert.match(html, /x-crucix-token/); + assert.match(html, /SET TOKEN/); +}); + test('server dashboard shell does not embed an operational snapshot', () => { const html = readFileSync(new URL('../dashboard/public/jarvis.html', import.meta.url), 'utf8'); assert.match(html, /let D = createDashboardShellData\(\);/);