Compare commits
24 Commits
3069114ffd
...
codex/prod
| Author | SHA1 | Date | |
|---|---|---|---|
| c159c83a07 | |||
|
|
a1d415e449 | ||
|
|
0f5f9c5f91 | ||
| 096544f6e6 | |||
|
|
5a3dbc6252 | ||
| 9f2083a324 | |||
| dd08ecaf27 | |||
|
|
bc354e7bc5 | ||
| 1c2b48f588 | |||
|
|
a590bf62c2 | ||
| 6a9918bc98 | |||
|
|
4448f5931b | ||
| 9b15913049 | |||
|
|
331175ae3c | ||
| e288881c41 | |||
|
|
e4834cd3cd | ||
|
|
0fbd8640ca | ||
|
|
09df127e06 | ||
|
|
c102017b16 | ||
|
|
eefc1a4c77 | ||
|
|
1423dca199 | ||
|
|
2025ae09db | ||
|
|
446076cb84 | ||
|
|
d7df2e4aee |
@@ -10,6 +10,9 @@ STALE_ALERT_COOLDOWN_MINUTES=60
|
|||||||
DASHBOARD_URL=
|
DASHBOARD_URL=
|
||||||
TERMINAL_ACTIONS_ENABLED=true
|
TERMINAL_ACTIONS_ENABLED=true
|
||||||
SWEEP_TOKEN=
|
SWEEP_TOKEN=
|
||||||
|
SSE_HEARTBEAT_INTERVAL_MS=25000
|
||||||
|
TERMINAL_ACTION_RATE_LIMIT_WINDOW_MS=60000
|
||||||
|
TERMINAL_ACTION_RATE_LIMIT_MAX=10
|
||||||
BRIEF_VERBOSITY=standard
|
BRIEF_VERBOSITY=standard
|
||||||
|
|
||||||
# LLM layer
|
# LLM layer
|
||||||
|
|||||||
36
README.md
36
README.md
@@ -134,6 +134,9 @@ STALE_ALERT_COOLDOWN_MINUTES=60
|
|||||||
DASHBOARD_URL=https://intelligence.example.internal
|
DASHBOARD_URL=https://intelligence.example.internal
|
||||||
TERMINAL_ACTIONS_ENABLED=true
|
TERMINAL_ACTIONS_ENABLED=true
|
||||||
SWEEP_TOKEN=
|
SWEEP_TOKEN=
|
||||||
|
SSE_HEARTBEAT_INTERVAL_MS=25000
|
||||||
|
TERMINAL_ACTION_RATE_LIMIT_WINDOW_MS=60000
|
||||||
|
TERMINAL_ACTION_RATE_LIMIT_MAX=10
|
||||||
BRIEF_VERBOSITY=standard
|
BRIEF_VERBOSITY=standard
|
||||||
|
|
||||||
LLM_PROVIDER=openrouter
|
LLM_PROVIDER=openrouter
|
||||||
@@ -185,9 +188,22 @@ LLM_MODEL=your-model
|
|||||||
|
|
||||||
For Pangolin or another reverse proxy, forward HTTP traffic to `intelligence-terminal:3117` (or the `PORT` you set). Missing API keys do not crash sweeps; affected sources are reported as degraded in `/api/health`.
|
For Pangolin or another reverse proxy, forward HTTP traffic to `intelligence-terminal:3117` (or the `PORT` you set). Missing API keys do not crash sweeps; affected sources are reported as degraded in `/api/health`.
|
||||||
|
|
||||||
When data remains stale past `STALE_DATA_MAX_AGE_MINUTES`, the server sends an operator alert through configured Telegram/Discord channels after failed or degraded sweep attempts. `STALE_ALERT_COOLDOWN_MINUTES` prevents repeated stale alerts from spamming every refresh interval. Set `DASHBOARD_URL` to the Pangolin/public URL you want included in those alerts.
|
#### Terminal Action Exposure
|
||||||
|
|
||||||
The dashboard Terminal Actions panel can trigger `status`, `sweep`, and `brief` through `/api/action`. Leave `TERMINAL_ACTIONS_ENABLED=true` for a private home-server deployment. For an internet-exposed deployment, set `SWEEP_TOKEN` and pass it through trusted automation, or set `TERMINAL_ACTIONS_ENABLED=false` to disable browser-triggered actions. If you protect actions with `SWEEP_TOKEN`, the browser can send it from `localStorage.crucix_sweep_token`.
|
`POST /api/action` and `POST /api/sweep` can trigger operational actions such as manual sweeps. The dashboard has a **SET TOKEN** control that stores your `SWEEP_TOKEN` in browser local storage and sends it as the `x-crucix-token` header; do not put action tokens in URLs.
|
||||||
|
|
||||||
|
Recommended settings:
|
||||||
|
|
||||||
|
| Deployment | Settings |
|
||||||
|
| --- | --- |
|
||||||
|
| Private local machine | `NODE_ENV=development`, optional `SWEEP_TOKEN`, optional `TERMINAL_ACTIONS_ENABLED=true`. Localhost can run actions without a token for development. |
|
||||||
|
| Private LAN / Dockge | Set a strong `SWEEP_TOKEN`, keep `TERMINAL_ACTIONS_ENABLED=true`, expose only to trusted clients. |
|
||||||
|
| Pangolin-authenticated reverse proxy | Set a strong `SWEEP_TOKEN`, keep Pangolin auth in front, use the dashboard **SET TOKEN** flow once per browser. |
|
||||||
|
| Public internet | Do not expose Terminal Actions directly. If exposure is unavoidable, require `SWEEP_TOKEN`, keep proxy authentication enabled, lower `TERMINAL_ACTION_RATE_LIMIT_MAX`, and monitor server audit logs. |
|
||||||
|
|
||||||
|
Action endpoints reject cross-origin POST origins, apply a small in-memory per-IP rate limit, and write sanitized audit lines without logging the token.
|
||||||
|
|
||||||
|
When data remains stale past `STALE_DATA_MAX_AGE_MINUTES`, the server sends an operator alert through configured Telegram/Discord channels after failed or degraded sweep attempts. `STALE_ALERT_COOLDOWN_MINUTES` prevents repeated stale alerts from spamming every refresh interval. Set `DASHBOARD_URL` to the Pangolin/public URL you want included in those alerts.
|
||||||
|
|
||||||
#### Memory And Prediction Loop
|
#### Memory And Prediction Loop
|
||||||
|
|
||||||
@@ -219,6 +235,22 @@ Retention, backup, and privacy expectations:
|
|||||||
- Do not commit `runs/` or `.env`. API credentials stay in `.env`; memory stores derived observations, not secrets.
|
- Do not commit `runs/` or `.env`. API credentials stay in `.env`; memory stores derived observations, not secrets.
|
||||||
- If you expose the dashboard through a reverse proxy, protect Terminal Actions and memory queries behind your normal authentication boundary.
|
- If you expose the dashboard through a reverse proxy, protect Terminal Actions and memory queries behind your normal authentication boundary.
|
||||||
|
|
||||||
|
#### Reverse Proxy SSE
|
||||||
|
|
||||||
|
The dashboard receives live sweep updates from `GET /events` using Server-Sent Events. The server sends `retry: 10000` reconnect guidance and lightweight heartbeat comments every `SSE_HEARTBEAT_INTERVAL_MS` milliseconds so reverse proxies do not close an otherwise idle stream between 15-minute sweeps.
|
||||||
|
|
||||||
|
Recommended proxy settings:
|
||||||
|
|
||||||
|
| Proxy | Setting |
|
||||||
|
| --- | --- |
|
||||||
|
| Pangolin / Traefik-style frontends | Keep response streaming enabled and set idle timeouts above `SSE_HEARTBEAT_INTERVAL_MS`. |
|
||||||
|
| Nginx | Disable proxy buffering for `/events`, keep `proxy_read_timeout` above the heartbeat interval, and preserve `Connection: keep-alive`. |
|
||||||
|
| Cloudflare-style proxies | Keep the heartbeat below common idle cutoffs; the default 25s is intentionally conservative. |
|
||||||
|
|
||||||
|
If you raise the heartbeat interval, keep it shorter than the lowest idle timeout in the proxy chain.
|
||||||
|
|
||||||
|
`/api/metrics` includes network health grouped by host and source/provider. Source modules should use `safeFetch(url, { source: 'SourceName' })`; when omitted, the shared helper infers a stable provider bucket from the URL host instead of grouping normal source traffic under `unknown`. Raw fetch exceptions are documented in [Source Fetch Instrumentation](docs/source-fetch-instrumentation.md).
|
||||||
|
|
||||||
#### Scenario Watchlist
|
#### Scenario Watchlist
|
||||||
|
|
||||||
Intelligence Terminal can track operator hypotheses across sweeps with a runtime scenario file at `runs/scenarios.json`. On first run, the server creates three disabled starter examples:
|
Intelligence Terminal can track operator hypotheses across sweeps with a runtime scenario file at `runs/scenarios.json`. On first run, the server creates three disabled starter examples:
|
||||||
|
|||||||
@@ -10,6 +10,44 @@ const fetchMetrics = {
|
|||||||
recent: [],
|
recent: [],
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const SOURCE_BY_HOST = [
|
||||||
|
[/api\.bls\.gov$/i, 'BLS'],
|
||||||
|
[/api\.fred\.stlouisfed\.org$/i, 'FRED'],
|
||||||
|
[/api\.eia\.gov$/i, 'EIA'],
|
||||||
|
[/api\.gdeltproject\.org$/i, 'GDELT'],
|
||||||
|
[/api\.weather\.gov$/i, 'NOAA'],
|
||||||
|
[/api\.open-notify\.org$/i, 'OpenNotify'],
|
||||||
|
[/opensky-network\.org$/i, 'OpenSky'],
|
||||||
|
[/firms\.modaps\.eosdis\.nasa\.gov$/i, 'FIRMS'],
|
||||||
|
[/api\.acleddata\.com$/i, 'ACLED'],
|
||||||
|
[/api\.reliefweb\.int$/i, 'ReliefWeb'],
|
||||||
|
[/receiverbook\.de$/i, 'KiwiSDR'],
|
||||||
|
[/safecast\.org$/i, 'Safecast'],
|
||||||
|
[/api\.patentsview\.org$/i, 'PatentsView'],
|
||||||
|
[/api\.trade\.gov$/i, 'Comtrade'],
|
||||||
|
[/api\.usaspending\.gov$/i, 'USASpending'],
|
||||||
|
[/api\.telegram\.org$/i, 'Telegram'],
|
||||||
|
[/oauth\.reddit\.com$/i, 'Reddit'],
|
||||||
|
[/reddit\.com$/i, 'Reddit'],
|
||||||
|
[/api\.bsky\.app$/i, 'Bluesky'],
|
||||||
|
[/api\.yahoo\.com$/i, 'YahooFinance'],
|
||||||
|
[/query\d?\.finance\.yahoo\.com$/i, 'YahooFinance'],
|
||||||
|
[/api\.cloudflare\.com$/i, 'CloudflareRadar'],
|
||||||
|
[/api\.opensanctions\.org$/i, 'OpenSanctions'],
|
||||||
|
[/home\.treasury\.gov$/i, 'Treasury'],
|
||||||
|
[/fiscaldata\.treasury\.gov$/i, 'Treasury'],
|
||||||
|
[/who\.int$/i, 'WHO'],
|
||||||
|
];
|
||||||
|
|
||||||
|
export function inferFetchSource(url) {
|
||||||
|
let host = 'unknown';
|
||||||
|
try { host = new URL(url).host.toLowerCase(); } catch { return 'unknown'; }
|
||||||
|
for (const [pattern, source] of SOURCE_BY_HOST) {
|
||||||
|
if (pattern.test(host)) return source;
|
||||||
|
}
|
||||||
|
return host;
|
||||||
|
}
|
||||||
|
|
||||||
function metricBucket(map, key) {
|
function metricBucket(map, key) {
|
||||||
if (!map[key]) map[key] = { requests: 0, ok: 0, failed: 0, bytes: 0, lastStatus: null, lastError: null, lastMs: 0 };
|
if (!map[key]) map[key] = { requests: 0, ok: 0, failed: 0, bytes: 0, lastStatus: null, lastError: null, lastMs: 0 };
|
||||||
return map[key];
|
return map[key];
|
||||||
@@ -38,7 +76,7 @@ export function getFetchMetrics() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export async function safeFetch(url, opts = {}) {
|
export async function safeFetch(url, opts = {}) {
|
||||||
const { timeout = 15000, retries = 1, headers = {}, source = undefined } = opts;
|
const { timeout = 15000, retries = 1, headers = {}, source = inferFetchSource(url) } = opts;
|
||||||
let lastError;
|
let lastError;
|
||||||
for (let i = 0; i <= retries; i++) {
|
for (let i = 0; i <= retries; i++) {
|
||||||
const started = Date.now();
|
const started = Date.now();
|
||||||
@@ -79,11 +117,11 @@ export async function safeFetch(url, opts = {}) {
|
|||||||
if (i < retries) await new Promise(r => setTimeout(r, 2000 * (i + 1)));
|
if (i < retries) await new Promise(r => setTimeout(r, 2000 * (i + 1)));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return { error: lastError?.message || 'Unknown error', source: url };
|
return { error: lastError?.message || 'Unknown error', source };
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function safeFetchText(url, opts = {}) {
|
export async function safeFetchText(url, opts = {}) {
|
||||||
const { timeout = 15000, retries = 1, headers = {}, source = undefined } = opts;
|
const { timeout = 15000, retries = 1, headers = {}, source = inferFetchSource(url) } = opts;
|
||||||
let lastError;
|
let lastError;
|
||||||
for (let i = 0; i <= retries; i++) {
|
for (let i = 0; i <= retries; i++) {
|
||||||
const started = Date.now();
|
const started = Date.now();
|
||||||
|
|||||||
@@ -26,7 +26,10 @@ export default {
|
|||||||
staleAlertCooldownMinutes: intEnv('STALE_ALERT_COOLDOWN_MINUTES', 60),
|
staleAlertCooldownMinutes: intEnv('STALE_ALERT_COOLDOWN_MINUTES', 60),
|
||||||
dashboardUrl: process.env.DASHBOARD_URL || null,
|
dashboardUrl: process.env.DASHBOARD_URL || null,
|
||||||
sweepToken: process.env.SWEEP_TOKEN || null,
|
sweepToken: process.env.SWEEP_TOKEN || null,
|
||||||
terminalActionsEnabled: boolEnv('TERMINAL_ACTIONS_ENABLED', true),
|
terminalActionsEnabled: boolEnv('TERMINAL_ACTIONS_ENABLED', !!process.env.SWEEP_TOKEN || process.env.NODE_ENV !== 'production'),
|
||||||
|
terminalActionRateLimitWindowMs: intEnv('TERMINAL_ACTION_RATE_LIMIT_WINDOW_MS', 60_000),
|
||||||
|
terminalActionRateLimitMax: intEnv('TERMINAL_ACTION_RATE_LIMIT_MAX', 10),
|
||||||
|
sseHeartbeatIntervalMs: intEnv('SSE_HEARTBEAT_INTERVAL_MS', 25000),
|
||||||
|
|
||||||
llm: {
|
llm: {
|
||||||
provider: process.env.LLM_PROVIDER || null, // anthropic | openai | gemini | codex | openrouter | minimax | mistral | ollama | grok
|
provider: process.env.LLM_PROVIDER || null, // anthropic | openai | gemini | codex | openrouter | minimax | mistral | ollama | grok
|
||||||
|
|||||||
@@ -432,6 +432,7 @@ let terminalOutput = 'Ready. Live data is loaded from /api/data in server mode.'
|
|||||||
let terminalBusy = false;
|
let terminalBusy = false;
|
||||||
let currentRegion = 'world';
|
let currentRegion = 'world';
|
||||||
let flatSvg, flatProjection, flatPath, flatG, flatZoom, flatW, flatH;
|
let flatSvg, flatProjection, flatPath, flatG, flatZoom, flatW, flatH;
|
||||||
|
const terminalActionTokenKey = 'crucix_sweep_token';
|
||||||
|
|
||||||
const layerTypeMap = {
|
const layerTypeMap = {
|
||||||
air: ['air'],
|
air: ['air'],
|
||||||
@@ -632,6 +633,7 @@ function renderTopbar(){
|
|||||||
const ts = new Date(D.meta.timestamp);
|
const ts = new Date(D.meta.timestamp);
|
||||||
const d = ts.toLocaleDateString('en-US',{month:'short',day:'numeric',year:'numeric'}).toUpperCase();
|
const d = ts.toLocaleDateString('en-US',{month:'short',day:'numeric',year:'numeric'}).toUpperCase();
|
||||||
const timeStr = ts.toLocaleTimeString('en-US',{hour:'2-digit',minute:'2-digit',hour12:true});
|
const timeStr = ts.toLocaleTimeString('en-US',{hour:'2-digit',minute:'2-digit',hour12:true});
|
||||||
|
const hasActionToken = !!getTerminalActionToken();
|
||||||
document.getElementById('topbar').innerHTML=`
|
document.getElementById('topbar').innerHTML=`
|
||||||
<div class="top-left">
|
<div class="top-left">
|
||||||
<span class="brand">CRUCIX MONITOR</span>
|
<span class="brand">CRUCIX MONITOR</span>
|
||||||
@@ -644,12 +646,26 @@ function renderTopbar(){
|
|||||||
<span class="meta-pill">${d} <span class="v">${timeStr}</span></span>
|
<span class="meta-pill">${d} <span class="v">${timeStr}</span></span>
|
||||||
<span class="meta-pill">${t('dashboard.sources','SOURCES')} <span class="v">${D.meta.sourcesOk}/${D.meta.sourcesQueried}</span></span>
|
<span class="meta-pill">${t('dashboard.sources','SOURCES')} <span class="v">${D.meta.sourcesOk}/${D.meta.sourcesQueried}</span></span>
|
||||||
${D.delta?.summary ? `<span class="meta-pill">${t('dashboard.delta','DELTA')} <span class="v">${D.delta.summary.direction==='risk-off'?'▲ '+t('dashboard.riskOff','RISK-OFF'):D.delta.summary.direction==='risk-on'?'▼ '+t('dashboard.riskOn','RISK-ON'):'◆ '+t('dashboard.mixed','MIXED')}</span></span>` : ''}
|
${D.delta?.summary ? `<span class="meta-pill">${t('dashboard.delta','DELTA')} <span class="v">${D.delta.summary.direction==='risk-off'?'▲ '+t('dashboard.riskOff','RISK-OFF'):D.delta.summary.direction==='risk-on'?'▼ '+t('dashboard.riskOn','RISK-ON'):'◆ '+t('dashboard.mixed','MIXED')}</span></span>` : ''}
|
||||||
|
<button class="guide-btn" onclick="configureTerminalActionToken()" title="Configure SWEEP_TOKEN for protected terminal actions">${hasActionToken?'TOKEN SET':'SET TOKEN'}</button>
|
||||||
<button class="guide-btn" onclick="openGlossary()">${t('dashboard.guideBtn','What Signals Mean')}</button>
|
<button class="guide-btn" onclick="openGlossary()">${t('dashboard.guideBtn','What Signals Mean')}</button>
|
||||||
<span class="alert-badge">${t('dashboard.highAlert','HIGH ALERT')}</span>
|
<span class="alert-badge">${t('dashboard.highAlert','HIGH ALERT')}</span>
|
||||||
</div>`;
|
</div>`;
|
||||||
renderRegionControls();
|
renderRegionControls();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function getTerminalActionToken(){
|
||||||
|
return localStorage.getItem(terminalActionTokenKey) || localStorage.getItem('crucix_terminal_action_token') || '';
|
||||||
|
}
|
||||||
|
|
||||||
|
function configureTerminalActionToken(){
|
||||||
|
const next = window.prompt('Terminal action token (SWEEP_TOKEN). Leave empty to clear.', getTerminalActionToken());
|
||||||
|
if(next === null) return;
|
||||||
|
const clean = next.trim();
|
||||||
|
if(clean) localStorage.setItem(terminalActionTokenKey, clean);
|
||||||
|
else localStorage.removeItem(terminalActionTokenKey);
|
||||||
|
renderTopbar();
|
||||||
|
}
|
||||||
|
|
||||||
// === LEFT RAIL ===
|
// === LEFT RAIL ===
|
||||||
function layerMode(key){ return layerModes[key] || 'normal'; }
|
function layerMode(key){ return layerModes[key] || 'normal'; }
|
||||||
function layerModeLabel(key){ return layerMode(key) === 'focus' ? 'focused' : layerMode(key) === 'hidden' ? 'hidden' : 'normal'; }
|
function layerModeLabel(key){ return layerMode(key) === 'focus' ? 'focused' : layerMode(key) === 'hidden' ? 'hidden' : 'normal'; }
|
||||||
@@ -1592,6 +1608,12 @@ function renderLower(){
|
|||||||
|
|
||||||
async function runTerminalAction(action){
|
async function runTerminalAction(action){
|
||||||
if(terminalBusy) return;
|
if(terminalBusy) return;
|
||||||
|
let token = getTerminalActionToken();
|
||||||
|
if(!token && location.hostname !== 'localhost' && location.hostname !== '127.0.0.1'){
|
||||||
|
configureTerminalActionToken();
|
||||||
|
token = getTerminalActionToken();
|
||||||
|
if(!token) return;
|
||||||
|
}
|
||||||
terminalBusy = true;
|
terminalBusy = true;
|
||||||
terminalOutput = `> ${action}\nRunning...`;
|
terminalOutput = `> ${action}\nRunning...`;
|
||||||
renderRight();
|
renderRight();
|
||||||
@@ -1600,7 +1622,7 @@ async function runTerminalAction(action){
|
|||||||
method:'POST',
|
method:'POST',
|
||||||
headers:{
|
headers:{
|
||||||
'Content-Type':'application/json',
|
'Content-Type':'application/json',
|
||||||
...(localStorage.getItem('crucix_sweep_token') ? {'x-crucix-token': localStorage.getItem('crucix_sweep_token')} : {})
|
...(token ? {'x-crucix-token': token} : {})
|
||||||
},
|
},
|
||||||
body:JSON.stringify({action})
|
body:JSON.stringify({action})
|
||||||
});
|
});
|
||||||
@@ -1698,6 +1720,7 @@ function renderRight(){
|
|||||||
<button class="action-btn" ${terminalBusy?'disabled':''} onclick="runTerminalAction('brief')">Brief</button>
|
<button class="action-btn" ${terminalBusy?'disabled':''} onclick="runTerminalAction('brief')">Brief</button>
|
||||||
<button class="action-btn" ${terminalBusy?'disabled':''} onclick="runTerminalAction('memory')">Memory</button>
|
<button class="action-btn" ${terminalBusy?'disabled':''} onclick="runTerminalAction('memory')">Memory</button>
|
||||||
</div>
|
</div>
|
||||||
|
<button class="mini-btn" style="margin-bottom:8px" onclick="configureTerminalActionToken()">Configure token</button>
|
||||||
<div class="terminal-output">${terminalOutput.replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])).replace(/\n/g,'<br>')}</div>
|
<div class="terminal-output">${terminalOutput.replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])).replace(/\n/g,'<br>')}</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="g-panel right-signals">
|
<div class="g-panel right-signals">
|
||||||
|
|||||||
21
docs/source-fetch-instrumentation.md
Normal file
21
docs/source-fetch-instrumentation.md
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# Source Fetch Instrumentation
|
||||||
|
|
||||||
|
`safeFetch()` and `safeFetchText()` attribute requests to `/api/metrics.fetch.bySource`.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- Prefer passing an explicit `source` option from source modules when the call has a clear Crucix source name.
|
||||||
|
- If `source` is omitted, the shared helper infers a stable provider name from the request host.
|
||||||
|
- Unknown hosts fall back to the lowercase host instead of the old `unknown` bucket.
|
||||||
|
- Raw `fetch()` calls should be limited to cases where the shared helper cannot represent the protocol cleanly.
|
||||||
|
|
||||||
|
Current raw-fetch exceptions:
|
||||||
|
|
||||||
|
| Area | Reason |
|
||||||
|
| --- | --- |
|
||||||
|
| OAuth/session handshakes | Token exchange calls often need custom form bodies, credential headers, or status-specific diagnostics. |
|
||||||
|
| Bot and alert delivery | Telegram/Discord alert calls are outbound operator notifications, not intelligence source health. |
|
||||||
|
| LLM providers | Provider clients already track model/provider status separately from source fetch health. |
|
||||||
|
| Dashboard browser calls | Browser-side `/api/*` and asset fetches are UI behavior, not source provider health. |
|
||||||
|
|
||||||
|
When adding a new intelligence source, use `safeFetch(url, { source: 'SourceName' })` unless there is a documented exception.
|
||||||
176
server.mjs
176
server.mjs
@@ -41,6 +41,7 @@ let sweepStartedAt = null; // Timestamp when current/last sweep started
|
|||||||
let sweepInProgress = false;
|
let sweepInProgress = false;
|
||||||
const startTime = Date.now();
|
const startTime = Date.now();
|
||||||
const sseClients = new Set();
|
const sseClients = new Set();
|
||||||
|
const terminalActionBuckets = new Map();
|
||||||
const staleAlertState = {};
|
const staleAlertState = {};
|
||||||
|
|
||||||
// === Delta/Memory ===
|
// === Delta/Memory ===
|
||||||
@@ -292,7 +293,9 @@ app.get('/api/metrics', (req, res) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
app.get('/api/memory/search', (req, res) => {
|
app.get('/api/memory/search', (req, res) => {
|
||||||
if (!canRunTerminalAction(req)) return res.status(403).json({ error: 'Memory queries disabled or unauthorized' });
|
const guard = authorizeTerminalAction(req, res, 'memory:search');
|
||||||
|
if (!guard.ok) return;
|
||||||
|
auditTerminalAction(req, 'memory:search', 'ok');
|
||||||
res.json(intelligenceStore.queryMemory({
|
res.json(intelligenceStore.queryMemory({
|
||||||
q: req.query.q || '',
|
q: req.query.q || '',
|
||||||
limit: req.query.limit || 25,
|
limit: req.query.limit || 25,
|
||||||
@@ -300,7 +303,9 @@ app.get('/api/memory/search', (req, res) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
app.get('/api/memory/predictions', (req, res) => {
|
app.get('/api/memory/predictions', (req, res) => {
|
||||||
if (!canRunTerminalAction(req)) return res.status(403).json({ error: 'Memory queries disabled or unauthorized' });
|
const guard = authorizeTerminalAction(req, res, 'memory:predictions');
|
||||||
|
if (!guard.ok) return;
|
||||||
|
auditTerminalAction(req, 'memory:predictions', 'ok');
|
||||||
res.json(intelligenceStore.listPredictions({
|
res.json(intelligenceStore.listPredictions({
|
||||||
state: req.query.state || null,
|
state: req.query.state || null,
|
||||||
limit: req.query.limit || 25,
|
limit: req.query.limit || 25,
|
||||||
@@ -308,24 +313,33 @@ app.get('/api/memory/predictions', (req, res) => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
app.post('/api/sweep', express.json(), (req, res) => {
|
app.post('/api/sweep', express.json(), (req, res) => {
|
||||||
if (!canRunTerminalAction(req)) return res.status(403).json({ error: 'Terminal actions disabled or unauthorized' });
|
const guard = authorizeTerminalAction(req, res, 'sweep');
|
||||||
triggerSweep(res);
|
if (!guard.ok) return;
|
||||||
|
triggerSweepAction(req, res, 'sweep');
|
||||||
});
|
});
|
||||||
|
|
||||||
app.post('/api/action', express.json(), async (req, res) => {
|
app.post('/api/action', express.json(), (req, res) => {
|
||||||
if (!canRunTerminalAction(req)) return res.status(403).json({ error: 'Terminal actions disabled or unauthorized' });
|
const action = String(req.body?.action || req.body?.command || '').trim().toLowerCase();
|
||||||
const action = String(req.body?.action || req.query.action || '').toLowerCase();
|
const guard = authorizeTerminalAction(req, res, action || 'unknown');
|
||||||
|
if (!guard.ok) return;
|
||||||
|
|
||||||
if (action === 'status') {
|
if (action === 'status') {
|
||||||
return res.json({ ok: true, action, health: buildHealth() });
|
auditTerminalAction(req, 'status', 'ok');
|
||||||
|
return res.json({ ok: true, action, status: 'ok', health: buildHealth() });
|
||||||
}
|
}
|
||||||
|
|
||||||
if (action === 'brief') {
|
if (action === 'brief') {
|
||||||
if (!currentData) return res.status(503).json({ ok: false, action, error: 'No data yet — first sweep in progress' });
|
if (!currentData) {
|
||||||
return res.json({ ok: true, action, text: buildBrief(currentData) });
|
auditTerminalAction(req, 'brief', 'rejected', 'no_data');
|
||||||
|
return res.status(503).json({ ok: false, action, error: 'No data yet - first sweep in progress' });
|
||||||
|
}
|
||||||
|
auditTerminalAction(req, 'brief', 'ok');
|
||||||
|
const brief = buildBrief(currentData);
|
||||||
|
return res.json({ ok: true, action, status: 'ok', brief, text: brief });
|
||||||
}
|
}
|
||||||
|
|
||||||
if (action === 'memory') {
|
if (action === 'memory') {
|
||||||
|
auditTerminalAction(req, 'memory', 'ok');
|
||||||
return res.json({
|
return res.json({
|
||||||
ok: true,
|
ok: true,
|
||||||
action,
|
action,
|
||||||
@@ -335,11 +349,10 @@ app.post('/api/action', express.json(), async (req, res) => {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
if (action === 'sweep') {
|
if (action === 'sweep') return triggerSweepAction(req, res, 'action:sweep');
|
||||||
return triggerSweep(res);
|
|
||||||
}
|
|
||||||
|
|
||||||
res.status(400).json({ ok: false, error: 'Unknown action', actions: ['status', 'brief', 'memory', 'sweep'] });
|
auditTerminalAction(req, action || 'unknown', 'rejected', 'unknown_action');
|
||||||
|
return res.status(400).json({ ok: false, error: 'Unknown action', allowed: ['status', 'brief', 'memory', 'sweep'], actions: ['status', 'brief', 'memory', 'sweep'] });
|
||||||
});
|
});
|
||||||
|
|
||||||
// API: available locales
|
// API: available locales
|
||||||
@@ -357,10 +370,24 @@ app.get('/events', (req, res) => {
|
|||||||
'Cache-Control': 'no-cache',
|
'Cache-Control': 'no-cache',
|
||||||
'Connection': 'keep-alive',
|
'Connection': 'keep-alive',
|
||||||
'Access-Control-Allow-Origin': '*',
|
'Access-Control-Allow-Origin': '*',
|
||||||
|
'X-Accel-Buffering': 'no',
|
||||||
});
|
});
|
||||||
|
res.write('retry: 10000\n');
|
||||||
res.write('data: {"type":"connected"}\n\n');
|
res.write('data: {"type":"connected"}\n\n');
|
||||||
|
const heartbeatMs = Math.max(5000, config.sseHeartbeatIntervalMs || 25000);
|
||||||
|
const heartbeat = setInterval(() => {
|
||||||
|
try {
|
||||||
|
res.write(`: heartbeat ${new Date().toISOString()}\n\n`);
|
||||||
|
} catch {
|
||||||
|
clearInterval(heartbeat);
|
||||||
|
sseClients.delete(res);
|
||||||
|
}
|
||||||
|
}, heartbeatMs);
|
||||||
sseClients.add(res);
|
sseClients.add(res);
|
||||||
req.on('close', () => sseClients.delete(res));
|
req.on('close', () => {
|
||||||
|
clearInterval(heartbeat);
|
||||||
|
sseClients.delete(res);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
function broadcast(data) {
|
function broadcast(data) {
|
||||||
@@ -370,26 +397,114 @@ function broadcast(data) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function requestIp(req) {
|
||||||
|
return req.ip || req.socket?.remoteAddress || 'unknown';
|
||||||
|
}
|
||||||
|
|
||||||
|
function isLocalRequest(req) {
|
||||||
|
const remote = requestIp(req);
|
||||||
|
return remote === '::1'
|
||||||
|
|| remote === '127.0.0.1'
|
||||||
|
|| remote === '::ffff:127.0.0.1'
|
||||||
|
|| remote.startsWith('127.')
|
||||||
|
|| remote === 'localhost';
|
||||||
|
}
|
||||||
|
|
||||||
|
function sameOriginPost(req) {
|
||||||
|
const origin = req.get('origin');
|
||||||
|
if (!origin) return true;
|
||||||
|
try {
|
||||||
|
const originUrl = new URL(origin);
|
||||||
|
const host = req.get('host');
|
||||||
|
return host && originUrl.host === host;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function actionToken(req) {
|
||||||
|
return req.get('x-crucix-token') || req.body?.token || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function auditTerminalAction(req, action, outcome, detail = null) {
|
||||||
|
const suffix = detail ? ` detail=${detail}` : '';
|
||||||
|
console.log(`[Crucix][audit] terminal_action action=${action || 'unknown'} outcome=${outcome} ip=${requestIp(req)}${suffix}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function rateLimitTerminalAction(req, action) {
|
||||||
|
const now = Date.now();
|
||||||
|
const windowMs = Math.max(1000, config.terminalActionRateLimitWindowMs || 60_000);
|
||||||
|
const max = Math.max(1, config.terminalActionRateLimitMax || 10);
|
||||||
|
const key = `${requestIp(req)}:${action}`;
|
||||||
|
const bucket = terminalActionBuckets.get(key);
|
||||||
|
if (!bucket || now > bucket.resetAt) {
|
||||||
|
terminalActionBuckets.set(key, { count: 1, resetAt: now + windowMs });
|
||||||
|
return { ok: true };
|
||||||
|
}
|
||||||
|
bucket.count += 1;
|
||||||
|
if (bucket.count > max) {
|
||||||
|
return { ok: false, retryAfterSeconds: Math.ceil((bucket.resetAt - now) / 1000) };
|
||||||
|
}
|
||||||
|
return { ok: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
function authorizeTerminalAction(req, res, action) {
|
||||||
|
const rate = rateLimitTerminalAction(req, action);
|
||||||
|
if (!rate.ok) {
|
||||||
|
auditTerminalAction(req, action, 'rejected', 'rate_limited');
|
||||||
|
res.set('Retry-After', String(rate.retryAfterSeconds));
|
||||||
|
res.status(429).json({ error: 'Too many terminal actions', retryAfterSeconds: rate.retryAfterSeconds });
|
||||||
|
return { ok: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!sameOriginPost(req)) {
|
||||||
|
auditTerminalAction(req, action, 'rejected', 'csrf_origin');
|
||||||
|
res.status(403).json({ error: 'Origin mismatch' });
|
||||||
|
return { ok: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
const local = isLocalRequest(req);
|
||||||
|
const token = actionToken(req);
|
||||||
|
if (!config.terminalActionsEnabled) {
|
||||||
|
auditTerminalAction(req, action, 'rejected', 'disabled');
|
||||||
|
res.status(403).json({ error: 'Terminal actions are disabled' });
|
||||||
|
return { ok: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config.sweepToken) {
|
||||||
|
if (token !== config.sweepToken) {
|
||||||
|
auditTerminalAction(req, action, 'rejected', 'invalid_token');
|
||||||
|
res.status(401).json({ error: 'Invalid terminal action token' });
|
||||||
|
return { ok: false };
|
||||||
|
}
|
||||||
|
return { ok: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!local) {
|
||||||
|
auditTerminalAction(req, action, 'rejected', 'missing_token');
|
||||||
|
res.status(403).json({ error: 'Terminal actions are local-only unless SWEEP_TOKEN is set' });
|
||||||
|
return { ok: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
return { ok: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
function triggerSweepAction(req, res, auditAction) {
|
||||||
|
if (sweepInProgress) {
|
||||||
|
auditTerminalAction(req, auditAction, 'rejected', 'already_running');
|
||||||
|
return res.status(409).json({ ok: true, status: 'already_running', sweepStartedAt });
|
||||||
|
}
|
||||||
|
auditTerminalAction(req, auditAction, 'accepted');
|
||||||
|
runSweepCycle().catch(err => console.error('[Crucix] API-triggered sweep failed:', err.message));
|
||||||
|
return res.status(202).json({ ok: true, status: 'accepted' });
|
||||||
|
}
|
||||||
|
|
||||||
function dataAgeMs() {
|
function dataAgeMs() {
|
||||||
const ts = currentData?.meta?.timestamp || lastSuccessfulSweepTime || lastSweepTime;
|
const ts = currentData?.meta?.timestamp || lastSuccessfulSweepTime || lastSweepTime;
|
||||||
const ms = ts ? Date.now() - new Date(ts).getTime() : null;
|
const ms = ts ? Date.now() - new Date(ts).getTime() : null;
|
||||||
return Number.isFinite(ms) ? ms : null;
|
return Number.isFinite(ms) ? ms : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
function canRunTerminalAction(req) {
|
|
||||||
const remote = req.ip || '';
|
|
||||||
const local = remote.includes('127.0.0.1') || remote === '::1' || remote === '::ffff:127.0.0.1';
|
|
||||||
const token = req.get('x-crucix-token') || req.query.token || req.body?.token;
|
|
||||||
if (config.sweepToken) return token === config.sweepToken;
|
|
||||||
return Boolean(config.terminalActionsEnabled || local);
|
|
||||||
}
|
|
||||||
|
|
||||||
function triggerSweep(res) {
|
|
||||||
if (sweepInProgress) return res.status(409).json({ ok: true, status: 'already_running', sweepStartedAt });
|
|
||||||
runSweepCycle().catch(err => console.error('[Crucix] API-triggered sweep failed:', err.message));
|
|
||||||
return res.status(202).json({ ok: true, status: 'accepted' });
|
|
||||||
}
|
|
||||||
|
|
||||||
function getLLMStatus() {
|
function getLLMStatus() {
|
||||||
if (!config.llm.provider) return { state: 'disabled' };
|
if (!config.llm.provider) return { state: 'disabled' };
|
||||||
if (!llmProvider) return { state: 'misconfigured', provider: config.llm.provider };
|
if (!llmProvider) return { state: 'misconfigured', provider: config.llm.provider };
|
||||||
@@ -433,7 +548,8 @@ function buildHealth() {
|
|||||||
llm: getLLMStatus(),
|
llm: getLLMStatus(),
|
||||||
telegramEnabled: !!(config.telegram.botToken && config.telegram.chatId),
|
telegramEnabled: !!(config.telegram.botToken && config.telegram.chatId),
|
||||||
discordEnabled: !!(config.discord?.botToken || config.discord?.webhookUrl),
|
discordEnabled: !!(config.discord?.botToken || config.discord?.webhookUrl),
|
||||||
terminalActionsEnabled: Boolean(config.terminalActionsEnabled || config.sweepToken),
|
terminalActionsEnabled: config.terminalActionsEnabled,
|
||||||
|
terminalActionsTokenRequired: !!config.sweepToken,
|
||||||
refreshIntervalMinutes: config.refreshIntervalMinutes,
|
refreshIntervalMinutes: config.refreshIntervalMinutes,
|
||||||
language: currentLanguage,
|
language: currentLanguage,
|
||||||
memory: intelligenceStore.status(),
|
memory: intelligenceStore.status(),
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import test from 'node:test';
|
import test from 'node:test';
|
||||||
import assert from 'node:assert/strict';
|
import assert from 'node:assert/strict';
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
import { safeFetch, safeFetchText, getFetchMetrics } from '../apis/utils/fetch.mjs';
|
import { safeFetch, safeFetchText, getFetchMetrics, inferFetchSource } from '../apis/utils/fetch.mjs';
|
||||||
import { formatStaleAlert, shouldSendStaleAlert } from '../lib/stale-alerts.mjs';
|
import { formatStaleAlert, shouldSendStaleAlert } from '../lib/stale-alerts.mjs';
|
||||||
|
|
||||||
test('safeFetch reports HTML as degraded JSON response', async () => {
|
test('safeFetch reports HTML as degraded JSON response', async () => {
|
||||||
@@ -101,6 +101,42 @@ test('safeFetchText returns text and byte count', async () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('safeFetch attributes unlabelled requests to a stable provider source', async () => {
|
||||||
|
const originalFetch = globalThis.fetch;
|
||||||
|
globalThis.fetch = async () => ({
|
||||||
|
ok: true,
|
||||||
|
status: 200,
|
||||||
|
headers: { get: () => 'application/json' },
|
||||||
|
text: async () => '{"observations":[]}',
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const data = await safeFetch('https://api.fred.stlouisfed.org/fred/series/observations?series_id=VIXCLS', { retries: 0 });
|
||||||
|
assert.deepEqual(data, { observations: [] });
|
||||||
|
const bucket = getFetchMetrics().bySource.FRED;
|
||||||
|
assert.ok(bucket.requests >= 1);
|
||||||
|
assert.equal(bucket.lastStatus, 200);
|
||||||
|
} finally {
|
||||||
|
globalThis.fetch = originalFetch;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('inferFetchSource returns provider names and host fallback', () => {
|
||||||
|
assert.equal(inferFetchSource('https://api.bls.gov/publicAPI/v2/timeseries/data/CPI'), 'BLS');
|
||||||
|
assert.equal(inferFetchSource('https://query1.finance.yahoo.com/v8/finance/chart/%5EGSPC'), 'YahooFinance');
|
||||||
|
assert.equal(inferFetchSource('https://unknown.example.test/path'), 'unknown.example.test');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('SSE endpoint sends reconnect guidance and clears heartbeat timer', () => {
|
||||||
|
const server = readFileSync(new URL('../server.mjs', import.meta.url), 'utf8');
|
||||||
|
const config = readFileSync(new URL('../crucix.config.mjs', import.meta.url), 'utf8');
|
||||||
|
assert.match(config, /sseHeartbeatIntervalMs/);
|
||||||
|
assert.match(server, /retry: 10000\\n/);
|
||||||
|
assert.match(server, /setInterval\(\(\) =>/);
|
||||||
|
assert.match(server, /: heartbeat/);
|
||||||
|
assert.match(server, /clearInterval\(heartbeat\)/);
|
||||||
|
assert.match(server, /X-Accel-Buffering/);
|
||||||
|
});
|
||||||
|
|
||||||
test('intelligence store defines durable memory and prediction lifecycle tables', () => {
|
test('intelligence store defines durable memory and prediction lifecycle tables', () => {
|
||||||
const store = readFileSync(new URL('../lib/intelligence-store.mjs', import.meta.url), 'utf8');
|
const store = readFileSync(new URL('../lib/intelligence-store.mjs', import.meta.url), 'utf8');
|
||||||
assert.match(store, /CREATE TABLE IF NOT EXISTS events/);
|
assert.match(store, /CREATE TABLE IF NOT EXISTS events/);
|
||||||
@@ -122,6 +158,25 @@ test('server exposes memory-backed query APIs and dashboard memory action', () =
|
|||||||
assert.match(html, /runTerminalAction\('memory'\)/);
|
assert.match(html, /runTerminalAction\('memory'\)/);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('terminal action endpoints avoid URL tokens and include hardening gates', () => {
|
||||||
|
const server = readFileSync(new URL('../server.mjs', import.meta.url), 'utf8');
|
||||||
|
assert.match(server, /app\.post\('\/api\/action'/);
|
||||||
|
assert.match(server, /app\.post\('\/api\/sweep'/);
|
||||||
|
assert.match(server, /x-crucix-token/);
|
||||||
|
assert.match(server, /sameOriginPost/);
|
||||||
|
assert.match(server, /rateLimitTerminalAction/);
|
||||||
|
assert.match(server, /auditTerminalAction/);
|
||||||
|
assert.doesNotMatch(server, /req\.query\.token/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('dashboard exposes token configuration flow without devtools edits', () => {
|
||||||
|
const html = readFileSync(new URL('../dashboard/public/jarvis.html', import.meta.url), 'utf8');
|
||||||
|
assert.match(html, /configureTerminalActionToken/);
|
||||||
|
assert.match(html, /crucix_sweep_token/);
|
||||||
|
assert.match(html, /x-crucix-token/);
|
||||||
|
assert.match(html, /SET TOKEN/);
|
||||||
|
});
|
||||||
|
|
||||||
test('server dashboard shell does not embed an operational snapshot', () => {
|
test('server dashboard shell does not embed an operational snapshot', () => {
|
||||||
const html = readFileSync(new URL('../dashboard/public/jarvis.html', import.meta.url), 'utf8');
|
const html = readFileSync(new URL('../dashboard/public/jarvis.html', import.meta.url), 'utf8');
|
||||||
assert.match(html, /let D = createDashboardShellData\(\);/);
|
assert.match(html, /let D = createDashboardShellData\(\);/);
|
||||||
|
|||||||
Reference in New Issue
Block a user