[Feature] Source Plugin Manifest and Per-Source Scheduling #30

New Issue

MrSphay · 2026-05-17T12:26:25Z

MrSphay commented

2026-05-17 12:26:25 +00:00

Summary

Introduce a source manifest layer so each data source declares metadata, credentials, category, expected freshness, dashboard surfaces, and preferred polling interval.

Why This Matters

The source list has grown beyond a simple hardcoded sweep. Some feeds need frequent polling, some should be slower, some require keys, and some are expensive or rate-limited. A manifest makes the system easier to extend and operate.

Scope

Bug fix: No
New source: No
Dashboard/UI change: Small diagnostics change
LLM/provider change: No
Other: Source architecture improvement

Proposed Behavior

Each source exports or is paired with metadata: id, display name, category, auth requirements, env vars, default timeout, default interval, stale threshold, and dashboard layer hints.
The orchestrator uses the manifest to decide what to run every sweep versus less frequently.
/api/health exposes manifest-based diagnostics, including missing credentials and expected freshness.
Source docs can be generated or checked against manifest entries.

Acceptance Criteria

Existing sources continue to work during migration.
The first implementation can support a compatibility mode for hardcoded imports.
Tests cover manifest validation, missing env var reporting, source interval decisions, and stale threshold behavior.
README and docs/sources/README.md describe how to add a source using the manifest.

Maintenance Impact

This is a structural change and should be done incrementally. It will reduce long-term cost for new source additions but must avoid a broad rewrite in one PR.

Additional Context

This gives future colleagues a cleaner path to add new providers without editing multiple unrelated files by hand.

## Summary Introduce a source manifest layer so each data source declares metadata, credentials, category, expected freshness, dashboard surfaces, and preferred polling interval. ## Why This Matters The source list has grown beyond a simple hardcoded sweep. Some feeds need frequent polling, some should be slower, some require keys, and some are expensive or rate-limited. A manifest makes the system easier to extend and operate. ## Scope - Bug fix: No - New source: No - Dashboard/UI change: Small diagnostics change - LLM/provider change: No - Other: Source architecture improvement ## Proposed Behavior - Each source exports or is paired with metadata: id, display name, category, auth requirements, env vars, default timeout, default interval, stale threshold, and dashboard layer hints. - The orchestrator uses the manifest to decide what to run every sweep versus less frequently. - `/api/health` exposes manifest-based diagnostics, including missing credentials and expected freshness. - Source docs can be generated or checked against manifest entries. ## Acceptance Criteria - Existing sources continue to work during migration. - The first implementation can support a compatibility mode for hardcoded imports. - Tests cover manifest validation, missing env var reporting, source interval decisions, and stale threshold behavior. - README and `docs/sources/README.md` describe how to add a source using the manifest. ## Maintenance Impact This is a structural change and should be done incrementally. It will reduce long-term cost for new source additions but must avoid a broad rewrite in one PR. ## Additional Context This gives future colleagues a cleaner path to add new providers without editing multiple unrelated files by hand.

MrSphay added the enhancement research product labels 2026-05-17 12:26:25 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: MrSphay/intelligence-terminal#30