Skip to main content
Glama

Spoor

Spoor tells an AI agent what changed in your services, and where it started, in a few hundred tokens.

CI License asinha07/spoor MCP server

Website: telespoor.com

Spoor is an open-source MCP server and CLI for your observability backends: the VictoriaMetrics stack, Prometheus, Loki, Tempo and New Relic. When something breaks, an agent asks Spoor "what changed in the last 30 minutes?" and gets back a short, ranked answer across metrics, logs and traces, with the service where the failure started at the top and its echoes in other services labelled as such.

Spoor (n.): the track or trail left by an animal. Spoor follows the trail a failure leaves in your telemetry back to where it started.

$ spoor what-changed --since 30m
{
  "summary": "11 change(s) in 3 services. Strongest: payment — Failures start here, in \"POST /charge\" (100% of 100 failed traces). Also: payment — Span \"POST /charge\" errors 0.2% → 43.7% (229 spans). … Errors not significantly up (not new) in: product-catalog, recommendation, ad.",
  "findings": [
    {"service": "payment",  "signal": "traces",  "kind": "error_origin",    "score": 0.99, "title": "Failures start here, in \"POST /charge\" (100% of 100 failed traces)"},
    {"service": "payment",  "signal": "logs",    "kind": "new_log_pattern", "score": 0.94, "title": "New error log pattern: \"charge failed: card processor returned <N> for order <N>\""},
    {"service": "payment",  "signal": "metrics", "kind": "error_rate_up",   "score": 0.92, "title": "Error rate up: 0.2% → 40.5%", "onset": "2026-10-16T09:50:00Z"},
    {"service": "checkout", "signal": "logs",    "kind": "spiking_log_pattern", "score": 0.74, "title": "Error log pattern spiking: \"call to payment.Charge failed with status <N>\" — originates in payment"}
  ],
  "coverage": {"metrics": "ok", "logs": "ok", "traces": "ok"}
}

Findings carry evidence links to the query behind them (when the token budget allows), so a human can check the agent's work in one click.

Why use it

In a microservice failure, the broken service and every caller of it all light up. An agent with only raw query tools has to decide what to query, compare against "normal" itself, page through results and work out which service is the cause and which are echoes. It burns tool calls and tokens, and often blames the loudest symptom or an error that was always there.

Spoor does that work in plain, deterministic Go and hands the agent the conclusion's evidence:

  • Cross-signal: request rate, error rate, latency, CPU, memory and consumer lag; new and spiking errors and error volume from logs; per-span errors and latency, all for the same window against the same baseline.

  • Built for real logs: messages that differ only by IDs, values, stack traces or log prefixes count as one error; JSON errors are read by their message and code fields; services can use different level fields or put the level in the text; spoor doctor finds your fields and writes the config.

  • Cause vs. echo: trace trees show where failures start and where latency is added. Callers that merely suffer are demoted and labelled — originates in payment.

  • Steady errors called out: errors that were already there are listed as steady_errors, "not a new problem", so agents stop blaming them.

  • Bounded output: every tool takes budget_tokens (default 800).

  • Honest about gaps: coverage says when a backend failed or had no data, the baseline was empty, the error filter matched nothing, or lines lack a service or level field. "No change" never stands in for "can't see".

  • No LLM inside, read-only, redacted: same input, same output; only query endpoints are ever called; every text field is scrubbed of emails, tokens, IPs and generated IDs before it leaves.

Measured

The benchmark runs Claude Code headless on six incidents (five faults and a no-fault control), five times each, once with the official VictoriaMetrics MCP servers alone and once with Spoor added. On Claude Haiku, 60 isolated sessions:

Official Victoria MCP servers

+ Spoor

Correct root cause, fault scenarios

19/25

25/25

Correct answer, all scenarios incl. "nothing is wrong"

23/30

30/30

Median tool calls per investigation

24

10

Median time

55 s

23 s

Total cost

$0.60

$0.19 (−68%)

Synthetic incidents and one small model, so treat it as a first data point.

On real production logs (about 25 services, mixed formats, read-only replay): the digest was 99.8% smaller than the raw evidence it replaces, with zero candidates against four from v0.1.1, and its baseline counts matched independent queries exactly. It was not faster than four hand-written queries (28 s vs 26 s), and accuracy on real incidents is not yet measured.

Method, per-scenario results, the production replay and caveats: docs/benchmark.md.

Related MCP server: mnm-incident-investigator

Quickstart

Requires Go 1.25+, Docker and jq. About five minutes.

git clone https://github.com/asinha07/spoor && cd spoor
make it-up          # empty VictoriaMetrics, VictoriaLogs, VictoriaTraces in Docker
make demo-data      # six synthetic incidents; ground truth in scenarios/out/
make build

T=scenarios/out/payment-failure.truth.json
./bin/spoor what-changed --since $(jq -r .since $T) --until $(jq -r .until $T) --pretty

Connect it to Claude Code:

claude mcp add spoor \
  -e SPOOR_METRICS_URL=http://localhost:8428 \
  -e SPOOR_LOGS_URL=http://localhost:9428 \
  -e SPOOR_TRACES_URL=http://localhost:10428 \
  -- "$PWD/bin/spoor" serve

Run it from the spoor directory (or add -s user to register it everywhere), then ask Claude: "Something is failing in the shop. What changed, and where did it start?"

On your own backends, start with spoor doctor --since 15m: it checks what Spoor can see and suggests config. Other clients, Grafana-only access and the rest: Getting started.

Backends

Each signal picks its backend with type:, and you can mix them:

Metrics

Logs

Traces

VictoriaMetrics stack

VictoriaMetrics

VictoriaLogs

VictoriaTraces

Grafana stack

Prometheus, Mimir, Thanos

Loki

Tempo

New Relic

Transaction events (NRQL)

Log events

Span events

metrics: {type: prometheus, url: http://prometheus:9090}
logs:    {type: loki, url: http://loki:3100}
traces:  {type: tempo, url: http://tempo:3200}

Setup and what differs per backend (Tempo statistics are sampled; New Relic span changes need the service's metrics to agree): Backends.

Tools

Tool

Returns

what_changed

Ranked changes across all signals, with cause/echo attribution, for one service or all. Start here.

new_errors

Error log patterns that are new or spiking, with counts, first-seen time and a redacted example.

trace_diff

Span names that got slower or started failing, with an example trace ID.

list_services

Services in the window and which signals have data for each.

Inputs: service, since, until, baseline (previous, 1d, 7d), budget_tokens. Full reference: docs/tools.md.

How it works

flowchart LR
    Agent[AI agent / CLI] -->|MCP| S[Spoor]
    S -->|"PromQL / NRQL"| M[(VictoriaMetrics, Prometheus, New Relic)]
    S -->|"LogsQL / LogQL / NRQL"| L[(VictoriaLogs, Loki, New Relic)]
    S -->|"LogsQL / TraceQL / NRQL"| T[(VictoriaTraces, Tempo, New Relic)]
    S -->|"ranked digest ≤ budget"| Agent

For a window and a baseline (by default, the six windows just before it), Spoor scores every metric with a robust z-score, compares error-log patterns with a Poisson test, compares span error rates and p95s, walks failed traces to the span where each failure started, boosts changes confirmed by more than one signal, and trims the result to the token budget. Details: architecture · scoring and attribution.

Run it for a team

spoor serve --http :8080 --auth-token … serves MCP over streamable HTTP with bearer-token auth, /healthz and Prometheus /metrics. Ready-made, hardened manifests:

Security, sizing, monitoring and upgrades: Production guide.

Verified on the OpenTelemetry Demo

On the OpenTelemetry Demo v3.1.0 (about 20 real services), a 50% payment failure came back as "payment: failures start here, in charge", with checkout and frontend errors labelled "originates in payment" (732 tokens); the adHighCpu fault came back as the single finding "ad: CPU up 0.07 → 3.97 cores" (192 tokens). Reproduce it: deploy/demo/.

Documentation

Getting started

Install, try, connect an agent

Production

Kubernetes, Docker, systemd, security, monitoring

Backends

VictoriaMetrics stack, Prometheus, Loki, Tempo, New Relic

Configuration

Every option and recipes for common setups

Tools

Inputs and response format

Architecture · How it works

Design and scoring

Benchmark · Comparison

Why use it, and what else exists

Troubleshooting · FAQ

When something looks wrong

Status and limitations

Early (pre-1.0). Tool output fields may still change between minor versions; see the changelog.

  • The scoring was developed and benchmarked on the VictoriaMetrics stack. Prometheus, Loki and Tempo pass the same scenario tests against real containers; New Relic was checked on one real account. Tempo statistics are sampled, and New Relic span changes are reported only when service metrics agree; see backends.

  • Cause/echo attribution needs traces; without them Spoor still ranks metric and log changes.

  • Attribution reads at most 50,000 span rows per read; on very busy systems a trace can be cut.

  • The benchmark uses synthetic incidents written by the author and one model; treat it as a first data point, and replay your own past incidents with bench/replay.py before relying on it.

  • Metrics and traces are queried per service; on large estates pass service when you know it.

More in how-it-works.md.

Roadmap

Next: accuracy on labelled real incidents, "since the last deploy" windows, Datadog and Elasticsearch backends, exact Tempo statistics through TraceQL metrics. The full list, and what is deliberately out of scope: docs/roadmap.md.

Contributing

Issues and pull requests are welcome. make test needs nothing but Go; see CONTRIBUTING.md. Security issues: SECURITY.md.

License

Apache-2.0. See LICENSE.

Available Tools

4 tools
list_servicesList servicesB
Read-onlyIdempotent

Services seen in the window and which signals (metrics, logs, traces) have data for each.

NameRequiredDescriptionDefault
sinceNoWindow start: RFC 3339 time or a duration ago. Default 30m.
untilNoWindow end. Default now.

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds the meaningful behavioral detail that the result is window-scoped and reports which signals have data per service, but says nothing about pagination, empty results, or service identity semantics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single tight sentence with no filler, and the scope qualifier ("in the window") is front-loaded. It is slightly terse to the point of being a fragment, but nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-optional-parameter read tool with no output schema, the description usefully characterizes the return value (services plus their per-signal data coverage), which is the main thing the agent needs to know. Only minor gaps remain around result size and service naming conventions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and both since/until are documented with formats and defaults in the schema itself. The description only echoes the window concept without adding format or interaction detail, so the schema does the heavy lifting and baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific resource (services) and scopes it to a time window, plus discloses the per-service signal breakdown (metrics, logs, traces). It reads as a noun phrase rather than an explicit verb, but the listing intent is unambiguous and clearly distinct from siblings like new_errors or trace_diff.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to reach for this tool versus the siblings (new_errors, trace_diff, what_changed), nor any prerequisite or exclusion. The agent must infer that this is the inventory/discovery entry point rather than anything the description states.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

new_errorsNew errorsA
Read-onlyIdempotent

Error logs compared with a baseline: errors that are new or spiking (messages that differ only by IDs count as one error), services whose error volume rose beyond their traffic, each with counts, first-seen time and one redacted example line, plus services whose errors are steady.

NameRequiredDescriptionDefault
sinceNoWindow start: RFC 3339 time or a duration ago such as 30m or 2h. Default 30m.
untilNoWindow end: RFC 3339 time or a duration ago. Default now.
serviceNoService name. Omit to scan every service seen in the window.
baselineNoWhat to compare against: previous (the windows just before; default), 1d or 7d (same window that many days earlier).
budget_tokensNoUpper bound on the response size in tokens, 200 to 4000. Default 800.

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so safety is covered. The description adds real behavioral context beyond that: the dedup rule that messages differing only by IDs count as one error, that each entry carries counts, first-seen time and a redacted example line, and that steady services are also reported. Auth, rate limits and ordering are not disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence, but front-loaded with the core comparison and then the result contents in priority order; every clause (dedup rule, counts, first-seen, redaction, steady services) carries information. It reads as a run-on, which costs a point, but nothing is padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must carry the return contract — and it does, enumerating the three result groups and the per-entry fields. What is missing is edge behavior: what happens when budget_tokens truncates results, ordering, and the exact shape of 'steady' services versus the rising ones.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and all five parameters (since, until, service, baseline, budget_tokens) are fully documented in the schema with defaults and formats. The description reinforces the baseline-comparison concept but adds no syntax, units or format detail beyond what the schema already states, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the specific analysis performed (error logs compared against a baseline) and the three result categories: new/spiking errors, services with error volume rising beyond traffic, and steady-error services. It is clearly distinguishable from list_services and trace_diff, though it never names a sibling explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the output framing — an agent can infer this is the tool for spotting newly introduced or regressing errors versus a baseline. However, there is no explicit when-to-use guidance, no statement of when NOT to use it, and no routing against the sibling what_changed, which could plausibly overlap.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

trace_diffTrace diffA
Read-onlyIdempotent

Span names that got slower or started failing compared with a baseline: p95 and error rate before/after, each span's share of the added latency, and an example trace ID.

NameRequiredDescriptionDefault
sinceNoWindow start: RFC 3339 time or a duration ago such as 30m or 2h. Default 30m.
untilNoWindow end: RFC 3339 time or a duration ago. Default now.
serviceNoService name. Omit to scan every service seen in the window.
baselineNoWhat to compare against: previous (default), 1d or 7d.
operationNoLimit to one span name, e.g. 'POST /checkout'.
budget_tokensNoUpper bound on the response size in tokens, 200 to 4000. Default 800.

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the safe read-only, idempotent, non-destructive profile, so the bar is low. The description goes beyond them by disclosing the exact result shape (p95 and error rate before/after, per-span latency share, an example trace ID), which is valuable since no output schema exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One dense sentence with a front-loaded subject and a colon-delimited list of returned fields; nothing is wasted. It is a noun phrase rather than a verb-first statement, which slightly weakens immediate readability but not materially.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden and does so well, and all six parameters are fully documented in the schema with annotations covering the safety profile. The only real gap is the absence of routing guidance among sibling tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter is already documented with formats, defaults, and examples. The description only restates the baseline concept and adds no syntax or constraints beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific analysis: span names that regressed in latency or error rate versus a baseline, plus the returned metrics. It clearly identifies the resource (spans) and operation (diff against a baseline), though it never names the siblings it should be preferred over.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the phrase 'compared with a baseline' and the regression framing, which tells an agent this is for finding performance/error regressions. However, it offers no explicit when-to-use rules and does not distinguish itself from what_changed or new_errors, which an agent must infer.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

what_changedWhat changedA
Read-onlyIdempotent

Ranked list of what changed in a service (or all services) across metrics, logs and traces, compared with a baseline. Returns a one-paragraph summary, findings with before/after values, onset times and evidence links, and per-signal coverage. Start here.

NameRequiredDescriptionDefault
sinceNoWindow start: RFC 3339 time or a duration ago such as 30m or 2h. Default 30m.
untilNoWindow end: RFC 3339 time or a duration ago. Default now.
serviceNoService name. Omit to scan every service seen in the window.
baselineNoWhat to compare against: previous (the windows just before; default), 1d or 7d (same window that many days earlier).
budget_tokensNoUpper bound on the response size in tokens, 200 to 4000. Default 800.

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive, closed-world), so the bar is lower. The description adds real value by disclosing the return structure: a summary paragraph, ranked findings with before/after values and onset times, evidence links, and per-signal coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the core purpose before the return-value detail. The trailing imperative 'Start here.' is a useful orientation cue rather than waste, though it sits at the end rather than the front.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of describing returns and does so thoroughly (summary, findings, coverage). Combined with fully documented parameters and annotations covering safety, an agent has nearly everything needed, though explicit routing to sibling tools is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter (since, until, service, baseline, budget_tokens) is already documented in the schema. The description's mention of 'baseline' and '(or all services)' overlaps with what the schema already states, adding no syntax or format detail beyond it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('what changed in a service... across metrics, logs and traces') and scopes it precisely with the baseline comparison. The 'Start here' framing positions it as the entry-point tool, distinguishing it from the drill-down siblings new_errors and trace_diff.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Start here' gives clear context that this is the workflow entry point, and 'Omit to scan every service' clarifies scope behavior. It stops short of naming when NOT to use it or routing to new_errors/trace_diff explicitly, so it lacks the full when-not/alternatives treatment of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv0.2.1
    • First observedlist_services
    • First observednew_errors
    • First observedtrace_diff
    • First observedwhat_changed

TDQS

A3.6/5.0

Scored across 4 tools

Disambiguation4/5

Each tool maps to a distinct signal scope: list_services inventories services, new_errors is log-specific, trace_diff is trace-specific, and what_changed is the cross-signal ranked entry point. The only soft overlap is that what_changed subsumes findings from new_errors and trace_diff, but its 'start here' framing and ranked-summary output keep the boundary workable.

Naming Consistency3/5

All names use snake_case casing, but the conventions are mixed: list_services is verb_noun, new_errors is adjective_noun, trace_diff is noun_noun, and what_changed is a question-phrase. Readable and memorable, but there is no predictable verb_noun pattern an agent can rely on.

Tool Count5/5

Four tools for a baseline-diffing observability server is well-scoped: one inventory, one per signal (logs, traces), and one aggregator. Each tool clearly earns its place with no gratuitous or redundant entries.

Completeness4/5

The surface covers service inventory, log errors, trace regressions, and a unified change ranking, which handles the core diagnostic workflow. Gaps are minor: no metric-specific diff tool (metrics only surface inside what_changed), no raw log/span search, and no way to configure the comparison window or baseline.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    A
    maintenance
    Enables users to investigate infrastructure incidents in plain English, correlate observability and deploy data with runbooks, and get evidence-backed root-cause proposals with approval-gated remediation.
    5
    -
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables a local AI agent to investigate service incidents by querying read-only telemetry such as service health, metrics, logs, and traces, then produce verifiable assessments with evidence links and uncertainty.
    Apache 2.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to perform observability analysis across Prometheus-compatible metrics and logs without prior vendor knowledge, including signal discovery, entity topology resolution, root-cause analysis, correlation, and capacity forecasting through natural-language anchors.
    Apache 2.0
  • A
    license
    A
    quality
    A
    maintenance
    Enables diagnosing long AI-agent traces by identifying the most likely root-cause steps, so users can quickly find where an agent run went wrong.
    1
    54 PyPI
    MIT