Observability Replay
Connects to a Prometheus HTTP API backend to read service graph metrics, CPU and memory usage, restarts, rollouts, and alerts. Enables application scorecards, incident replay, service graph inspection, recent changes, and benchmark reports through read-only MCP tools.
Works as a Prometheus-compatible observability backend via the Prometheus HTTP API. Provides the same read-only scorecards, incident replay, service graph, alerts, changes, and benchmark capabilities from data available through Thanos.
Works as a Prometheus-compatible observability backend via the Prometheus HTTP API. Allows the MCP server to produce application scorecards, incident replays, service graph views, alert and change listings, and benchmark reports from VictoriaMetrics data.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Observability replay
An MCP server that gives an agent two views over any observability backend:
Application scorecards: how each application is doing, worst first, with the same measures on every cloud: request volume, p95 latency, error ratio against the error budget, CPU and memory used against what it requested, restarts, and whether its last deploy made things better or worse.
Incident replay: the service map animated over a time window, with traffic flowing between services, errors and latency spreading, and deploys and alerts marked on a timeline you can scrub.
It is built to be backend-agnostic. The agent and the view only see a neutral model; one adapter per observability tool translates into it.
Try it in one command
Needs Node 20 or newer.
npx observability-replay liveThen open http://127.0.0.1:8766/. With no backend configured it plays a scripted incident, so you can see all three pages before connecting anything.
Related MCP server: kafka-sentinel-mcp
Try it from a checkout
Needs Node 20 or newer (nvm use picks 22).
npm ci
npm run build
xdg-open dist/apps-preview.html # scorecards
xdg-open dist/preview.html # incident replay
xdg-open dist/benchmark-preview.html # benchmark reportOn macOS use open instead of xdg-open; on Windows, open the files from
the file explorer.
The preview plays a scripted incident: a bad payments deploy on GCP slows
payments, makes checkout on AWS time out, and reaches the gateway, until a
rollback. Add #at=0.4 to the replay URL to open paused 40% of the way
through, or #app=inventory to the scorecards URL to open one application.
The demo also scripts three resource stories: payments hits its CPU limit
while failing, inventory leaks memory until it restarts, and cart uses a
tenth of the CPU it requests.
Use it from an MCP client
{
"mcpServers": {
"observability": {
"command": "npx",
"args": ["-y", "observability-replay"],
"env": { "OBS_BACKEND": "demo" }
}
}
}From a checkout, use "command": "node" with
"args": ["/absolute/path/to/observability-replay/dist/server.js"] instead.
The animated view appears in clients that support MCP Apps. In other clients the tools still work and return text.
Tool | What it returns |
| The scorecard page, plus each application's status and findings as text |
| The animated replay, plus a text summary of what left baseline and in what order |
| Services, calls, and each one's latest rate, error ratio and p95 latency |
| Alerts that fired in the window |
| Deploys and rollbacks in the window |
| The benchmark report page, plus the grade and each check that needs a look as text |
| Which backend is connected and what it can supply |
Every tool is read-only. Windows are limited to 5 to 360 minutes and at most 240 frames, so no request can pull an unbounded amount of history.
Connect Prometheus
Works with anything that speaks the Prometheus HTTP API (Prometheus, Thanos, Mimir, VictoriaMetrics, managed Prometheus).
Variable | Meaning |
| Select the adapter |
| Base URL, e.g. |
| Optional bearer token |
| Optional label matchers for every graph query, e.g. |
| Metric prefix, default |
| Latency histogram without |
| Skip the rollout lookup if kube-state-metrics is not scraped |
| Skip CPU and memory if cAdvisor and kube-state-metrics are not scraped |
| Label on container metrics that equals the service name, default |
| Optional label matchers for every container and rollout query, e.g. |
The service graph is read from the traces_service_graph_request_* metrics that
the OpenTelemetry Collector's servicegraph connector and Tempo's
metrics-generator emit. Rollouts are read from kube-state-metrics
(kube_replicaset_created); a rollback reuses an old ReplicaSet, so rollbacks
are not seen. CPU, memory, throttling, restarts and replicas come from cAdvisor
and kube-state-metrics, joined to a service by name: the container name must
equal the service name in the traces, or set OBS_PROM_WORKLOAD_LABEL. A workload
that has container metrics but appears in no trace still gets a scorecard,
scored on its resources alone.
During development, the adapter queries were exercised against a live Prometheus 3.16
with kube-state-metrics 2.20 and the OpenTelemetry Collector 0.165 service
graph connector, on a k3d cluster running a traced application: the service
graph, an outage found by both views, CPU, memory, replicas and rollouts.
Alerts are read from ALERTS but no alert fired during those live runs.
These are development observations, not a compatibility guarantee; the included
tests use fixtures and a local mock server.
Two things the live runs taught, both handled:
A failure counter is born non-zero. The first failures on an edge create the series already at, say, 12, and
rate()sees no increase. Run Prometheus with--enable-feature=created-timestamp-zero-ingestionand have the exporter send creation times (the Collector'senable_open_metrics: true), or the start of an incident is lost.Memory looks like it spikes during a rollout if usage is summed over pods that are already gone. Usage is counted only for pods that kube-state-metrics still lists.
A Prometheus for a local cluster
deploy/monitoring/monitoring.yaml puts Prometheus, kube-state-metrics and
an OpenTelemetry Collector in their own monitoring namespace. Prometheus
scrapes each kubelet's cAdvisor. Applications send spans to the Collector
(http://otel-collector.monitoring.svc:4318, OTLP over HTTP); it turns them
into the service graph metrics and stores no traces. Its filter lists the
span sources and paths to leave out of the graph (SDK-level spans that
repeat a network call, readiness probes); adjust it to your application.
kubectl apply -f deploy/monitoring/monitoring.yaml
export OBS_BACKEND=prometheus
export OBS_PROM_URL=http://127.0.0.1:9090
export OBS_PROM_WORKLOAD_MATCHERS='namespace="example"' # your applications' namespace
npm startIn a separate terminal, run kubectl -n monitoring port-forward svc/prometheus 9090:9090. The manifest is for a disposable local cluster: it uses a NodePort,
unauthenticated endpoints, cluster-wide read access, and local storage. Do not
expose it to the public internet. The optional deploy/monitoring/prom-url.sh <k3d-node-container> helper prints the NodePort URL on hosts that can reach
Docker container addresses.
Watch a backend live
npx observability-replay live (or npm run live in a checkout) serves the three pages on the connected backend's own data,
on this machine only (127.0.0.1), so you can leave the applications page
open while something happens: a deploy, a load test, a pod being killed.
npm run build
export OBS_BACKEND=prometheus OBS_PROM_URL=http://127.0.0.1:9090
npm run live -- --minutes 15 --refresh 10 # then open http://127.0.0.1:8766/Each page load reads the backend again, and the applications page re-reads
every --refresh seconds (0 turns that off) and keeps the open application
selected. --minutes is the window each page covers and --port changes the
port. The pages are read-only and have no sign-in, which is why they are bound
to the loopback address; do not put them behind a public proxy.
What the scorecard measures
Objectives are set once for the whole fleet: OBS_SLO_AVAILABILITY (percent,
default 99.9) and OBS_SLO_P95_MS (default 500).
Measure | Definition |
Errors | Failed requests over all requests in the window, weighted by traffic |
Burn rate | Errors divided by the error budget (100% minus the availability objective); 1× is exactly on budget, 14.4× or more is critical |
Latency | Minutes of the window with p95 above the objective |
CPU of request | CPU used divided by CPU requested, mean and peak |
CPU throttled | Peak share of CPU periods throttled; 25% or more is a warning |
Memory of limit | Peak memory in use divided by the memory limit; 90% or more is a warning |
Restarts | Container restarts in the window; three or more is critical |
Change effect | p95 and errors between the previous change and this one, against between this one and the next; "regressed" means p95 more than doubled (and +100 ms) or errors more than tripled (and +2 points) |
A regression is critical while nothing has been rolled out since, and a warning once a later deploy or rollback has followed it. Advice to lower an oversized CPU request is a note and never makes an application unhealthy. Cost is not measured yet.
The benchmark
One fixed standard for a healthy system, the same on every cloud and backend.
What is measured, the bars and the grading are defined in src/benchmark.ts
and cannot be configured, so two results are always comparable; changing any
of them makes a new version of the standard (now 1.0).
Area | Check | Pass | Warn |
Availability | Requests succeed | 99.9% or more | 99% or more |
Latency | p95 stays under 500 ms | over for at most 1% of the window | at most 10% |
Resources | A memory limit is set | set | |
Resources | A CPU request is set | set | |
Resources | Memory stays clear of its limit | peak 80% or less | 90% or less |
Resources | CPU is not throttled | 5% of periods or less | 25% or less |
Stability | Containers do not restart | 0 | 1 or 2 |
Changes | Deploys do not make things worse | none regressed | one did, and was followed |
Resilience | Requests succeed under steady load | 99.9% or more | 99% or more |
Resilience | p95 latency under steady load | 500 ms or less | 1 s or less |
Resilience | Recovers after a failure | within 60 s | within 300 s |
Resilience | The failure is visible in monitoring | the backend shows errors |
The first eight are scored per application, the resilience four once per test. A pass is 1 point, a warn half, a fail none. Each area's score is the share of points over the checks that could be measured, and the overall score is the mean of the areas, so a system with many applications is not graded mostly on its resource settings. A is 90 or more, B 75, C 60, D 40. Two rules keep the grade honest: a check that could not be measured counts neither way, and no grade is given if fewer than half could be measured; and a failed "requests succeed" or "recovers after a failure" check holds the grade at C.
For an active test, first follow the plan setup
and create benchmark/plans/example.local.json. Only run plans you have reviewed:
their fault and cleanup fields execute shell commands with your permissions.
Two ways to run it, both reading the backend from the same OBS_* variables:
npm run benchmark -- observe --minutes 30
npm run benchmark -- test --plan benchmark/plans/example.local.json --compare results/earlier.jsonObserve only reads: it grades what the system did in the window. It is also the
benchmarkMCP tool.Test is active, so point it at a test system only. It sends a steady load for 60 s, runs the plan's fault command, removes the fault after 30 s, and watches for up to 300 s until ten requests in a row succeed again. The timings are fixed by the standard; a plan supplies only the URL to load, the rate, and the two shell commands that break something and put it back. In test mode the per-application availability and latency checks are left out, because the window was broken on purpose.
Each run is saved under results/ as JSON and as a report page (one
self-contained HTML file that opens in any browser, light or dark, and prints).
The page leads with the grade, then the score by area, what to fix with the
widest problems first, the test timeline, and every application against every
check. --compare adds what changed since an earlier result. To redraw the
page for a saved result:
npm run benchmark -- report results/example-run.json --compare results/earlier.jsondist/benchmark-preview.html is the same page on the scripted demo incident.
How it is put together
src/model.ts the neutral model (services, edges, samples, alerts, changes)
src/adapters/types.ts the Adapter interface and its capability flags
src/adapters/demo.ts scripted incident
src/adapters/prometheus.ts
src/replay.ts bounded time ranges, baseline/onset detection, the bundle, the text summary
src/scorecard.ts the scorecard definitions, findings and text summary
src/benchmark.ts the benchmark standard, its checks, scoring and comparison
src/bench.ts the benchmark command line: observe, and the active test
src/server.ts the MCP tools and the view resources
view/ the three pages, each bundled into one HTML fileThree rules the code holds to:
The agent never draws the data. It chooses the window and the focus; the adapter runs the queries; the view draws only what came back. Motion is interpolated between samples, but every printed number is a real sample.
Every view shows its source queries, in the backend's own language, so a number can be checked in the source tool.
A gap stays a gap. Missing samples are drawn as "no data", never filled.
"Off baseline" means beyond 3× the median of the first fifth of the window (with a floor of +100 ms latency or +2 points of error ratio). A service with no samples at the start of the window has its errors judged against zero. An incident already under way when the window opens will therefore not be detected; widen the window.
Adding a backend
Implement Adapter from src/adapters/types.ts, set its capability flags
honestly, and register it in src/adapters/index.ts. Nothing else changes.
Tests
npm testContributing and security
See CONTRIBUTING.md for setup and pull requests, and
SECURITY.md for the security model and vulnerability reporting.
Saved reports contain backend data; results/ and local benchmark plans are
excluded from Git. Review any report before sharing it.
License
Licensed under the Apache License 2.0.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Read-only MCP server for AIStatusDashboard status, incidents, metrics, and fallback recommendations.
Read-only health context MCP server for Iris users.
Read-only MCP access to a documented IT fleet: state, changes, posture. 15 tools.
Guarded MCP server for agent-readable business truth, provenance, readiness, and discovery.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceIn-cluster MCP server for read-only diagnostics of Grafana, Prometheus, Alertmanager, and Loki, enabling metric queries, alert listings, and log queries through Grafana datasource proxies with Kubernetes RBAC authentication.-
- AlicenseBqualityCmaintenanceRead-only MCP server for Kafka cluster health, consumer lag, partition state, and replay-readiness, enabling AI agents to diagnose streaming incidents without write access.83MIT
- AlicenseNot gradedqualityCmaintenanceMCP server providing read-only operational tools (logs, metrics, traces, service health, config) for troubleshooting an environment, with one exception for toggling chaos scenarios.MIT
- AlicenseNot gradedqualityDmaintenanceMCP server that provides guarded, audited, read-only access to ops tooling (alerts, metrics, logs, deploys, runbooks) and a triage agent that diagnoses incidents end-to-end with CI-verified root cause analysis.MIT