Cassandra's Hoard

Observability for the whole local AI stack on your PC. It watches every local service you run — the Faustus workspace and its test instances, llama-server, Ollama, ComfyUI, the Hoard Hub launcher and every Hoard app — and remembers what is up, what went down and when, what else changed at that same moment, what the logs said and what the GPUs were doing. When something stops at 04:00 and nobody knows why, Cassandra has the answer (or the most probable one). An assistant reaches the same data through MCP, so you can simply ask "why did Borges stop last night?".

Everything stays on the machine: one SQLite file, no accounts, no telemetry, no network beyond loopback health checks of your own services and, only if you list public websites, one request per site every few minutes.

Part of the Hoard family (see faustus-plugin.json).

What it does

• Registry of services, merged from three sources:

  • discovered apps: the Hoard Hub's own list (GET http://127.0.0.1:8810/api/apps, one source of truth for the whole family; CASSANDRA_HUB_REGISTRY=0 disables it, /api/status shows registry_source) and, when the hub is not there, every <root>/*/faustus-plugin.json (the same manifest the launcher reads). The manifest's app.health gives the health path and the expected service; data/url in the app folder wins over the manifest port;

  • built-in externals: Faustus (7000 main, 7001-7003 test, /api/health), llama-server (8081, helper 8082, /health), Ollama (11434, /api/tags), ComfyUI (8188-8191, /system_stats) and Hoard Hub (8810). An external that never answered shows as never seen, never as an incident;

  • your own services in data/services.json (id, name, url, health path, expected JSON, log paths, restart policy). An entry with the id of a discovered app or an external edits it.

• Poller: every CASSANDRA_POLL_S seconds (20 by default) all services are checked in parallel with short timeouts. The listening ports are read first (psutil), so a closed port is down at once (on Windows connecting to a closed loopback port takes ~1.5 s to fail). States: up, degraded (HTTP error or slower than CASSANDRA_SLOW_MS), foreign (another program answers on that port), down. A process that still listens on its port but does not answer is usually busy (a loaded PC, a long request): it stays degraded ("confirming") until it has not answered for CASSANDRA_HUNG_AFTER_S seconds (60), and only then is down with an incident and a notification; a busy spell that ends sooner opens nothing. Incidents of one service less than 30 minutes apart form a burst: cassandra.incident.opened carries burst_id (the first incident of the run) and burst_count, and the hub's notify rule sends one notification per burst. The pid behind each port, its start time and a hash of its command line are kept: a new pid on a service that stayed up is a restart event.

• Incidents: opened when a service goes from up to down/foreign (or its process is replaced), closed when it answers again. At open time a context is captured: the other services that changed within ±3 min (filled in by later polls), GPU memory just before and the peak of the last 3 min, the last lines of that service's logs, the process (pid, command line, alive or gone) and the machine's boot time. A probable cause is written in plain words from heuristics, for example:

  • "The machine restarted (boot at 03:58…): a reboot stops every service."

  • "Cassandra itself was not running between 01:10 and 07:30 (sleep, hibernation or shutdown?)…"

  • "3 services fell in the same minute (…) → machine-wide event (sleep, shutdown, GPU driver reset?)."

  • "GPU 1 memory jumped to 98% 40 s before → VRAM contention (another model loading?)."

  • "The log ends with a Traceback: “RuntimeError: …”." / "The log reports running out of memory: …"

  • "The process (pid 1234 python.exe) is still alive but does not answer: hung, overloaded or still loading."

• System pseudo-service: boot time and uptime; a boot-time change is recorded as a reboot event, and a long pause of Cassandra's own loop (sleep, hibernation, closed app) as a gap, shown striped on the lanes.

• GPU: nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu every poll (skipped silently when there is no NVIDIA driver).

• Logs: incremental tailing of each app's data/logs/*.log, the launcher's data/logs/<app>.log (the output of the apps it started), %LOCALAPPDATA%\Hoards\*.log, Cassandra's own data/logs/<id>.log (apps it restarted), each user service's log_paths and CASSANDRA_LOG_GLOBS. A file seen for the first time is read from its last 64 KB; rotation (a file that shrinks) is handled. Each line gets a time (parsed from the line when it has one) and a level (error/warning/info/debug heuristic). Retention: CASSANDRA_RETENTION_DAYS (14) and CASSANDRA_LOG_MAX_LINES.

• Audit trail (Hoard Link 0.4): the Hoard Hub keeps a rolling event bus every app posts to — one agent.call per tool the assistant ran, app milestones (scribe.transcript.done, links.watch.new…), hub actions (hub.backup.done, hub.rule.ran, hub.app.started). Cassandra mirrors it for good into bus_events (GET <hub>/api/events?since_id=, every 10 s, CASSANDRA_BUS=0 to disable, CASSANDRA_HUB_URL for the hub), so svc_why_down also shows what the assistant did in the three minutes around an incident, and audit_search / audit_stats answer "who called what, when, and what failed". Cassandra posts its own cassandra.incident.opened / closed events to the bus, so a hub rule can react to a service going down.

• Job incidents: every app reports its long work to the hub as <app>.job.queued|started|progress|done|failed (and the hub adds work.failed). When the mirrored bus carries a *.job.failed or work.failed, Cassandra opens an incident of kind job: service = the app, probable cause = the error text, context = the job (id, title, kind, url) and its own events from the bus. One incident per app and job kind stays open; another failure of the same kind is added to it and the newest error becomes the cause. A later *.job.done of the same app and kind closes it, and so does 24 hours without a new failure. The incidents page shows it with a job chip. It is announced as cassandra.incident.opened / closed with kind: "job", service_kind: "job" and to_state: "failed", so the hub rules that start an app again or notify about a dead app do not fire for it (the hub already notifies about the failed job itself). A failure older than 24 hours in a long bus history opens nothing. CASSANDRA_JOB_INCIDENTS=0 turns it off.

• Family agenda: GET /api/family/agenda (with the app's bearer token) answers the hub's Today list and calendar with the incidents that are still open (a service or site that is down, a job that failed and was not followed by a success) as items of kind incident, priority high, from the moment they opened until now. Closed incidents are history and are not listed.

• Secrets audit: secrets_audit walks every app folder Cassandra discovered: the agent token file (present, permissions), whether data/ is git-ignored, secret-looking files tracked by git (mcp-token, .env, *.key, databases), .env files. Read-only, no network.

• Restart policies (opt-in, per service): the policy's own command; or, for a discovered app while the launcher is up, POST http://127.0.0.1:8810/api/apps/<id>/start (/restart if it still runs); or the app's manifest launch_hint ({X_DIR}, {FAUSTUS_PYTHON} resolved like the launcher does). Never more than max_per_hour automatic restarts per service; never over a port held by another program; every attempt is recorded in the incident. A manual restart (UI button or svc_restart) is always allowed for a service that has a way to start.

• Public websites (data/sites.json, never in code): Cassandra watches your public sites from outside. Per site: HTTP (expected status, default 2xx/3xx; an optional keyword that must appear in the first 512 KB of the page, in which case redirects are followed), latency, TLS certificate (days left, issuer, hostname and chain validity, re-read hourly), DNS (the addresses; a change is recorded as an event, not an incident) and domain expiry from RDAP (IANA bootstrap, once a day per registrable domain; a TLD without RDAP shows unknown). A site is down after 2 consecutive failures (the first failure schedules one confirming check 60 s later) and up on the first success; the incident opens and closes like any service's, with a cause in plain words (timeout, DNS failure, TLS error, unexpected status, keyword missing). Warnings, each once: certificate at 21, 7 and 1 days, domain at 30, 7 and 1 days; a renewal is recorded. Bus events cassandra.site.down, .up, .cert_expiring and .domain_expiring; with Boop configured, the incident notifications plus the two expiry warnings (no site names in the text). Each check is one GET of the home page with the User-Agent Cassandra's Hoard site check/<version> (monitor run by the site owner), at most once per interval (default 5 min, 1 to 1440); a manual check of a site is spaced at least a minute from the previous one. The list is edited from the Panel card, with sites_watch, or by hand (picked up on the next read). CASSANDRA_SITES=0 turns it all off.

• What Faustus is running (faustus_farm, the Panel's Agents running card): one read-only call to Faustus's GET /api/farm/state (built once a second, with an ETag, so an unchanged picture is a 304). Every chat turn in flight is a row (title, kind, model, state, how long, progress) with the sub-agents it started indented under it; dispatched jobs show their workers, and a night shift, workflow runs and scheduled tasks in progress are listed too. Under the rows, one budget line: the window used against its pace line, GPU seconds today, whether the failure breaker is open and the provider cooldowns. The card refreshes every 4 s only while the Panel is visible, and says why when Faustus cannot be read (no token, refused, down, too old). Only names and states travel, never what anyone typed.

• UI (Spanish or English, automatic, switchable; dark theme): Panel (status grid grouped by kind with a 24 h lane per service, machine and GPU summary, the Agents running card, the Public websites card with its list editor, check now, restart), Incidents (filters by service, window or "around 04:00", open only; each incident expands to the explanation, what else changed, GPUs, process, log tail and actions), GPU (plain SVG chart of memory and load, peaks, free memory), Logs (search by words, service, level and time), Services (automatic-restart switch, max/hour and command per service, add or remove your own services, effective configuration). Installable as a PWA.

Related MCP server: MemoryOS VPS Guardian

Requirements

• Windows 10/11 (also Linux/macOS), Python 3.11+ (3.13 fine), Node 22 only to build the client.

• psutil (in requirements) for ports, pids and boot time. On macOS listing other processes' sockets may need extra permissions; Cassandra then falls back to HTTP-only checks.

• An NVIDIA driver (nvidia-smi) is optional: without it the GPU parts are simply empty.

Install and run

Windows:

git clone <this repo> CassandraHoard
cd CassandraHoard
python -m venv venv
venv\Scripts\pip install -r requirements.txt
npm.cmd install --include=dev
npx.cmd vite build
venv\Scripts\python -m cassandra_hoard

Linux / macOS:

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
npm install --include=dev && npx vite build
.venv/bin/python -m cassandra_hoard

Open http://127.0.0.1:5190. Put the repository next to the other Hoard apps (their parent folder is the default discovery root) or set CASSANDRA_ROOTS.

• python scripts/launch.py starts the app on a free port and opens the browser.

• python scripts/dev.py runs the API with reload plus the Vite dev server (proxying /api).

• python scripts/make_icon.py --src <Icons>/dragon-src.png regenerates the icon from the family dragon (needs requirements-icon.txt).

Configuration (environment)

Optional Boop incident notifications are disabled by default. See configuration, tests and delivery limits before enabling them.

data/services.json example:

{
  "services": [
    {"id": "whisper", "name": "Whisper server", "url": "http://127.0.0.1:9000", "health_path": "/health",
     "expect": {"status": "ok"}, "log_paths": ["C:/tools/whisper/logs/*.log"],
     "restart": {"enabled": true, "cmd": ["C:/tools/whisper/run.bat"], "cwd": "C:/tools/whisper", "max_per_hour": 3}}
  ],
  "policies": {"ollama": {"enabled": true, "cmd": "ollama serve", "max_per_hour": 2}}
}

data/sites.json example (the public websites; the id comes from the host, expect_status defaults to ["2xx","3xx"], interval_min to 5):

{
  "sites": [
    {"url": "https://example.com/", "name": "Shop", "keyword": "Add to cart", "interval_min": 5},
    {"url": "https://www.example.org/", "expect_status": ["200", "301-308"], "enabled": false}
  ]
}

API

GET /api/health ({"service": "cassandra-hoard", ...}), /api/status, /api/services, /api/services/{id}, /api/services/{id}/history, /api/lanes?hours=24, /api/incidents (since, until, at, window_min, service, open_only), /api/incidents/{id}, /api/logs (q, service, level, time window), /api/logs/sources, /api/gpu, /api/settings; POST /api/poll, POST /api/services (add/edit), DELETE /api/services/{id}, PUT /api/services/{id}/policy, POST /api/services/{id}/restart; the websites: GET /api/sites?hours=24, GET /api/sites/{id}/history, POST /api/sites (add/edit), POST /api/sites/check (check now), DELETE /api/sites/{id}; GET /api/faustus/farm (what Faustus is running now, or why it cannot be read), GET /api/agent/tools, POST /api/agent/call (Bearer token from data/mcp-token); the assistant panel: GET /api/copilot/info, GET /api/copilot/panel.js, /api/copilot/proxy, GET /api/copilot/launcher.js (see Faustus assistant panel). Times accept ISO (2026-09-24T04:00), a clock time (04:00 = the last 04:00) or an age (2h, 30m, 1d).

Faustus assistant panel

Every page has a floating Faustus button (bottom right) that opens the Faustus assistant in a side panel, with one conversation kept across reloads. The server side comes from the shared Hoard Link (install_spa mounts the relay and adds the launcher to the HTML: GET /api/copilot/info, GET /api/copilot/panel.js, /api/copilot/proxy, GET /api/copilot/launcher.js); the UI tells the panel what is on screen through window.hoardCopilot.context() (client/src/copilot-context.js). What each turn carries, beside the page route and title:

The filters that produced the results on screen go in view.filters. When the assistant finishes a turn the open view reloads in place (status, services, lanes, websites, the last search of Incidents, Logs and Audit); what is being typed is left alone. Website edits and restart policies are saved as patches: only the fields the user changed are sent, so a change made by the assistant in the meantime is not overwritten by the form.

Turn it off with HOARD_COPILOT=0 in the environment (or install_spa(app, STATIC_DIR, copilot=False)); the manifest declares the panel under x-family.copilot in faustus-plugin.json.

MCP tools

mcp_server.py is a stdio bridge (the shared catalogue bridge of Hoard Link): it fetches the catalogue from the app, proxies every call with the token, never opens the database, and starts the app itself when it is not answering (CASSANDRA_BRIDGE_AUTOSTART=0 turns that off). The token in data/mcp-token is created once and kept across restarts; starting the app a second time just says it is already running. The request guard, the agent routes (results capped at 20 KB), the single-page-app server and the database layer also come from the shared Hoard Link commons.

Tests

python -m pytest -q      # fake services (ASGI apps flipping up/down), fake clock, GPU and process table
npx vite build

Privacy

Everything is local: health checks go only to the URLs in the registry (loopback by default) and to the websites you list in data/sites.json (one GET of the home page per interval with a User-Agent that names Cassandra's Hoard, one TLS handshake an hour, one RDAP lookup per domain a day; CASSANDRA_SITES=0 turns all of it off), logs are read from your own disk and stored in data/cassandra.db, the two Faustus cards (waiting, running) read read-only routes of the loopback Faustus you configure and show names and states, never message text, nothing is sent anywhere and there is no telemetry. The API accepts only local origins (plus CASSANDRA_ALLOWED_HOSTS), and the agent routes need the token in data/mcp-token.

Limits (v1)

• Probable causes are heuristics over what Cassandra saw; when it was not running, it can only say the event happened inside that gap.

• The exit code of a process that Cassandra did not start is not available; it reports whether the process is gone or still alive.

• GPU sampling is NVIDIA-only (nvidia-smi).

• Websites are watched from this machine only: a failure of your own connection can look like every site being down (the probable cause says when DNS or the network itself failed). The confirming check after a first failure is the one request beyond the configured interval.

• Domain expiry depends on the registry publishing it over RDAP; some TLDs do not, and then the card shows no domain days. A failed RDAP lookup is retried after 6 hours.

• A CDN that rotates IP addresses shows up as DNS change events; they are informational.

License

MIT — see LICENSE.