<!-- mcp-name: io.github.mcpsmiths/tracehub-mcp -->

# tracehub-mcp

[![CI](https://github.com/mcpsmiths/tracehub-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/mcpsmiths/tracehub-mcp/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/mcpsmiths/tracehub-mcp/graph/badge.svg)](https://codecov.io/gh/mcpsmiths/tracehub-mcp)
[![PyPI](https://img.shields.io/pypi/v/tracehub-mcp.svg)](https://pypi.org/project/tracehub-mcp/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
[![mcpsmiths/tracehub-mcp MCP server](https://glama.ai/mcp/servers/mcpsmiths/tracehub-mcp/badges/score.svg)](https://glama.ai/mcp/servers/mcpsmiths/tracehub-mcp)
[![M8ven Score](https://m8ven.ai/badge/mcp/mcpsmiths/tracehub-mcp)](https://m8ven.ai/mcp/mcpsmiths/tracehub-mcp)

Also listed on the [official MCP registry](https://registry.modelcontextprotocol.io/v0/servers/io.github.mcpsmiths%2Ftracehub-mcp/versions) as `io.github.mcpsmiths/tracehub-mcp` (the registry's own `?search=` endpoint can surface an outdated version first; this exact-name endpoint always reflects the current `isLatest` release).

[![tracehub-mcp MCP server – quality and maintenance score on Glama](https://glama.ai/mcp/servers/mcpsmiths/tracehub-mcp/badges/card.svg)](https://glama.ai/mcp/servers/mcpsmiths/tracehub-mcp)

**Give your AI assistant a direct line into your observability backend.** tracehub-mcp is an MCP (Model Context Protocol) server that lets Claude, Cursor, Windsurf, Gemini CLI, or any MCP client query OpenTelemetry traces from your LLM/GenAI application and reason about them — find expensive calls, debug errors, compare model performance, track token usage — without you copy-pasting trace JSON into a chat window.

It speaks OpenTelemetry's `gen_ai.*` semantic conventions natively, so it understands prompts, completions, token usage, and finish reasons as first-class concepts, not just generic span attributes.

tracehub-mcp started as a fork of [traceloop/opentelemetry-mcp-server](https://github.com/traceloop/opentelemetry-mcp-server) (Apache 2.0) — full attribution and fork history are in [NOTICE](NOTICE). It's grown into an 8-backend, security-hardened server maintained independently under [mcpsmiths](https://github.com/mcpsmiths); see [What's Different From Upstream](#whats-different-from-upstream) below for the parts that are new here.

---

## Table of Contents

- [Quick Start](#quick-start)
- [Supported Backends](#supported-backends)
- [What's Different From Upstream](#whats-different-from-upstream)
- [Installation](#installation)
- [Configuration](#configuration)
- [Diagnostics](#diagnostics)
- [Security Considerations](#security-considerations)
- [Privacy Policy](#privacy-policy)
- [MCP Client Setup](#mcp-client-setup)
- [Tools Reference](#tools-reference)
- [Generic Filter System](#generic-filter-system)
- [Example Queries](#example-queries)
- [Common Workflows](#common-workflows)
- [Development](#development)
- [Troubleshooting](#troubleshooting)
- [Roadmap](#roadmap)
- [License](#license)
- [Support](#support)

---

## Quick Start

tracehub-mcp is on PyPI. No install step needed — `uvx` fetches and runs it in one shot:

```json
// claude_desktop_config.json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}
```

Or from Claude Code directly:

```bash
claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp
```

**That's it.** Ask your assistant: _"Show me traces with errors from the last hour."_

See [MCP Client Setup](#mcp-client-setup) for Cursor, Windsurf, VS Code, and Gemini CLI, and [Installation](#installation) for `pip`/`pipx`/from-source alternatives.

---

## Supported Backends

- **[Jaeger](https://www.jaegertracing.io/)** — local/self-hosted, the most common open-source trace backend. No auth required.
- **[Grafana Tempo](https://grafana.com/oss/tempo/)** — local or Grafana Cloud, TraceQL-native search.
- **[Traceloop](https://www.traceloop.com/)** — cloud LLM observability platform, API-key auth.
- **[Datadog](https://www.datadoghq.com/)** — cloud APM, requires an API key *and* an Application key.
- **[Sentry](https://sentry.io/)** — cloud or self-hosted, requires an auth token and an organization slug.
- **[AWS X-Ray](https://aws.amazon.com/xray/)** — SigV4-signed via `boto3`, requires an AWS region and standard AWS credentials (not an API key). Because OTel span attributes land in unindexed X-Ray segment *metadata* by default (only *annotations* are queryable), native server-side filtering is narrower here than for the other backends — see the [Backend Support Matrix](#backend-support-matrix).
- **[New Relic](https://newrelic.com/)** — NerdGraph GraphQL API, requires a User API key (not an Ingest/License key) and an account ID. **Built without a live account to verify against** — see the [New Relic setup section](#backend-specific-setup) below and the module docstring in [backends/newrelic.py](src/opentelemetry_mcp/backends/newrelic.py) for exactly which assumptions are unverified.
- **[Honeycomb](https://www.honeycomb.io/)** — Query Data API, requires a Configuration Key and a dataset slug. **Built without a live account to verify against**, and the research behind it found the programmatic Query Data API (the only way to actually run a query and get results back) is documented as Enterprise-plan exclusive — see the [Honeycomb setup section](#backend-specific-setup) below and the module docstring in [backends/honeycomb.py](src/opentelemetry_mcp/backends/honeycomb.py).

All eight implement the same `BaseBackend` interface, so every MCP tool works identically regardless of which one you point the server at. See [Configuration](#configuration) for per-backend setup.

---

## What's Different From Upstream

Upstream `opentelemetry-mcp-server` shipped Jaeger, Tempo, and Traceloop. tracehub-mcp adds **Datadog and Sentry** as full backends — not thin wrappers, but complete implementations of every tool (search, span search, trace hydration, service discovery, health checks). Along the way, all five backends — including the three inherited from upstream — were hardened to a consistent bar:

- **HTTPS-only enforcement** on cloud backends. Datadog and Sentry both refuse to start against a plain `http://` URL, because their auth is a bearer token / API+App key pair that has no business going out over plaintext.
- **Query-injection-safe escaping.** Every value spliced into a Datadog span-search query or a Sentry Discover query is escaped and exact-quoted; field *names* (which are less obviously untrusted, since they come from the MCP tool's `filters` parameter) are validated against an allowlist pattern before being spliced into the query string, closing off structural injection through a crafted field name.
- **Bounded pagination** on every backend that paginates via cursor (Datadog, Sentry) — a search stops at the requested `limit` or when the backend stops returning a continuation cursor, whichever comes first, so a single tool call can't degrade into an unbounded crawl.
- **Exact-ID re-verification.** Where a backend's search API can return neighbors instead of an exact match (notably Datadog's trace reconstruction from grouped spans), every result is re-checked against the exact ID that was asked for before being returned.
- **No fabricated data on malformed responses, in the backends we built.** Datadog and Sentry reject a span outright — rather than substituting a placeholder like `now()` for a missing timestamp or a literal `"unknown"` for a missing `service_name`/`operation_name` — since a fabricated value would silently corrupt trace ordering, duration aggregation, and any tool that groups by service or operation. (The three backends inherited from upstream — Jaeger, Tempo, Traceloop — predate this discipline and haven't been retrofitted; that's deliberate scope discipline, not an oversight, mirroring this project's own precedent of not reaching into shared/inherited code without full regression coverage for it.)

All of this is backed by **458 passing tests (2 skipped, zero regressions)**, a clean `ruff check` and `mypy --strict` run, and two rounds of adversarial CodeRabbit review on the new backends.

---

## Installation

tracehub-mcp is [on PyPI](https://pypi.org/project/tracehub-mcp/). Pick whichever of these your workflow already uses — they're equivalent.

### Option 1: `uvx` (no install step)

```bash
uvx tracehub-mcp --backend jaeger --url http://localhost:16686
```

This is what the [Quick Start](#quick-start) config above uses — `uv` fetches the package and runs the `tracehub-mcp` entry point in one shot, nothing left behind on disk between runs.

### Option 2: `pip` / `pipx`

```bash
pipx install tracehub-mcp
# or: pip install tracehub-mcp

tracehub-mcp --backend jaeger --url http://localhost:16686
```

### Option 3: Clone and run from source

```bash
git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync

uv run tracehub-mcp --backend jaeger --url http://localhost:16686
```

Use this if you're developing locally, want to pin to a specific commit, or want the dev tooling installed (`uv sync --group dev`).

### Option 4: Docker

```bash
docker run --rm -p 8000:8000 \
  -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://host.docker.internal:16686 \
  ghcr.io/mcpsmiths/tracehub-mcp:latest
```

Runs HTTP transport by default (the image's `CMD`); clients connect to `http://localhost:8000/mcp`. This is also the form to use for MCP clients whose config takes a `command`/`args` pair pointing at `docker` directly (Cursor, Windsurf) instead of a local binary.

**Prerequisites:** Python 3.11+, plus [uv](https://github.com/astral-sh/uv) for Options 1 and 3; Docker for Option 4.

---

## Configuration

Configuration comes from environment variables, CLI flags, or both. **Precedence: CLI arguments > environment variables > defaults.**

```bash
# .env (see .env.example)
BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686
```

```bash
# Equivalent via CLI flags
tracehub-mcp --backend jaeger --url http://localhost:16686
```

### All Configuration Options

| Variable                 | Type    | Default  | Description                                                          |
| ------------------------ | ------- | -------- | ---------------------------------------------------------------------|
| `BACKEND_TYPE`           | string  | `jaeger` | Backend type: `jaeger`, `tempo`, `traceloop`, `datadog`, `sentry`, `xray`, `newrelic`, or `honeycomb` |
| `BACKEND_URL`            | URL     | -        | Backend API endpoint (required; decorative-only placeholder for X-Ray) |
| `BACKEND_API_KEY`        | string  | -        | API key/auth token (required for Traceloop, Datadog, Sentry, New Relic, and Honeycomb; unused by X-Ray) |
| `BACKEND_APP_KEY`        | string  | -        | Application key (Datadog only, in addition to `BACKEND_API_KEY`)     |
| `BACKEND_TEMPO_INSTANCE_ID` | string | -     | Grafana Cloud stack/instance ID (Tempo only, enables Basic Auth in addition to `BACKEND_API_KEY`) |
| `BACKEND_SENTRY_ORG`     | string  | -        | Organization slug (required for Sentry)                             |
| `BACKEND_SENTRY_PROJECT` | string  | -        | Project slug (optional for Sentry, narrows queries to one project)   |
| `BACKEND_AWS_REGION`     | string  | -        | AWS region, e.g. `us-east-1` (required for X-Ray)                    |
| `BACKEND_NEWRELIC_ACCOUNT_ID` | string | -   | New Relic account ID (required for New Relic - NerdGraph queries are user-scoped, not account-scoped) |
| `BACKEND_HONEYCOMB_DATASET` | string | -     | Honeycomb dataset slug (required for Honeycomb - the Query Data API is dataset-scoped) |
| `BACKEND_ENVIRONMENTS`   | string  | `prd`    | Comma-separated environments (Traceloop only)                        |
| `BACKEND_TIMEOUT`        | float   | `30`     | Request timeout in seconds                                           |
| `SECONDARY_BACKEND_TYPE` / `SECONDARY_BACKEND_URL` / ... | string/URL | unset (opt-in) | A second, independently-configured backend for `correlate_trace` (cross-backend correlation) - every `BACKEND_*` variable above has a `SECONDARY_BACKEND_*` equivalent. Env-var only, no CLI flags. Unset means `correlate_trace` raises a clear error if called |
| `LOG_LEVEL`              | string  | `INFO`   | Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR` (`--log-level`)    |
| `MAX_TRACES_PER_QUERY`   | integer | `500`    | Server-wide ceiling (1-1000, `--max-traces-per-query`) - caps every tool's `limit` argument before it reaches a backend query, regardless of what the calling agent requests |
| `SLOW_REQUEST_THRESHOLD_MS` | float | unset  | Logs a WARNING for any backend request slower than this, independent of `LOG_LEVEL` (`--slow-request-threshold-ms`) |
| `MCP_TRANSPORT` / `MCP_HOST` / `MCP_PORT` | string/int | `stdio`/`0.0.0.0`/`8000` | Env-var equivalents of `--transport`/`--host`/`--port` |
| `MCP_INCLUDE_ARGS_IN_SPANS` | bool | `false` | Include tool call arguments/results as OTel span attributes when self-instrumentation is enabled below - off by default since they may contain sensitive data. Known credential shapes (Bearer tokens, `api_key=`/`secret=`/`password=`-style fields, AWS/GitHub/common vendor key prefixes) are redacted before export, but this is a pattern match, not a guarantee - trace_id/span_id and other legitimate trace data are deliberately left untouched (`--include-args-in-spans`) |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | URL | unset | Enables opt-in OTel self-instrumentation of tool calls when set; unset means zero overhead (no TracerProvider configured, no middleware registered) |
| `OTEL_SERVICE_NAME`      | string  | `tracehub-mcp` | Service name reported in self-instrumentation spans                |
| `QUERY_CACHE_TTL_SECONDS` | float  | unset  | Cache backend query results (`search_traces`/`search_spans`/`get_trace`/`list_services`/`get_service_operations`) for this many seconds, with in-flight request coalescing (unset: disabled, `--query-cache-ttl-seconds`) |
| `RATE_LIMIT_MAX_REQUESTS` | integer | `100` | HTTP transport only. Max requests per client IP per `RATE_LIMIT_WINDOW_SECONDS` (`--rate-limit-max-requests`, set to `0` to disable) |
| `RATE_LIMIT_WINDOW_SECONDS` | float | `60`  | HTTP transport only. Fixed window size in seconds for `RATE_LIMIT_MAX_REQUESTS` (`--rate-limit-window-seconds`) |
| `SHUTDOWN_DRAIN_SECONDS` | float   | `0.0`  | HTTP transport only. On shutdown, wait this many seconds inside the ASGI shutdown handler - after uvicorn has already finished draining in-flight connections - before closing the shared backend HTTP client (`--shutdown-drain-seconds`) |
| `BACKEND_CLOSE_TIMEOUT_SECONDS` | float | `5.0` | HTTP transport only. Abandon closing the shared backend HTTP client during shutdown if it does not finish within this many seconds - uvicorn places no timeout of its own around this wait (`--backend-close-timeout-seconds`) |
| `GRACEFUL_SHUTDOWN_TIMEOUT_SECONDS` | integer | `2` | HTTP transport only. uvicorn's own bound (whole seconds only) on waiting for in-flight connections/tasks to finish on shutdown before cancelling them (`--graceful-shutdown-timeout-seconds`) |

Every backend-related CLI flag has a matching env var (`--backend`/`BACKEND_TYPE`, `--url`/`BACKEND_URL`, `--api-key`/`BACKEND_API_KEY`, `--app-key`/`BACKEND_APP_KEY`, `--tempo-instance-id`/`BACKEND_TEMPO_INSTANCE_ID`, `--sentry-org`/`BACKEND_SENTRY_ORG`, `--sentry-project`/`BACKEND_SENTRY_PROJECT`, `--aws-region`/`BACKEND_AWS_REGION`, `--environments`/`BACKEND_ENVIRONMENTS`). `--disable-tools <name1,name2,...>` / `--enabled-tools <name1,name2,...>` (CLI-only, no env var) remove/allowlist tools for reduced-trust deployments - `--enabled-tools` is applied first, `--disable-tools` on top of whatever it kept. Run `tracehub-mcp --help` for the full list.

**Known third-party egress dependency:** the underlying FastMCP framework checks PyPI (`https://pypi.org/pypi/fastmcp/json`) for a newer FastMCP release once every 12 hours when it prints its startup banner - this is FastMCP's own behavior, not tracehub-mcp's, and unrelated to the OTel self-instrumentation above. It fails silently if there's no network access. For network-restricted/air-gapped deployments, disable it with `FASTMCP_CHECK_FOR_UPDATES=off`, or suppress the banner entirely with `FASTMCP_SHOW_SERVER_BANNER=false`.

### Backend-Specific Setup

<details>
<summary><b>Jaeger</b></summary>

```bash
BACKEND_TYPE=jaeger
BACKEND_URL=http://localhost:16686
```

No API key required. **`search_traces` and `search_spans_tool` both require a `service_name` parameter** — Jaeger's API is optimized for per-service queries, so querying across all services isn't supported. Discover service names first with `list_services`.

</details>

<details>
<summary><b>Grafana Tempo</b></summary>

```bash
BACKEND_TYPE=tempo
BACKEND_URL=http://localhost:3200
```

No API key required for a local/self-hosted install. Search uses [TraceQL](https://grafana.com/docs/tempo/latest/traceql/) under the hood; `service_name` is optional.

For **Grafana Cloud**-hosted Tempo, also set `BACKEND_TEMPO_INSTANCE_ID` to the stack's instance ID and `BACKEND_API_KEY` to a Cloud Access Policy token scoped to `traces:read` — Grafana Cloud requires Basic Auth (instance ID as username, token as password) instead of self-hosted Tempo's Bearer-token auth:

```bash
BACKEND_TYPE=tempo
BACKEND_URL=https://tempo-prod-XX-prod-XX-XXXX.grafana.net
BACKEND_TEMPO_INSTANCE_ID=your_stack_instance_id
BACKEND_API_KEY=your_cloud_access_policy_token
```

</details>

<details>
<summary><b>Traceloop</b></summary>

```bash
BACKEND_TYPE=traceloop
BACKEND_URL=https://api.traceloop.com
BACKEND_API_KEY=your_api_key_here
```

The API key encodes project information — the backend always uses a project slug of `"default"`, and Traceloop resolves the actual project/environment from the key itself.

</details>

<details>
<summary><b>Datadog</b></summary>

```bash
BACKEND_TYPE=datadog
# US site (default): https://api.datadoghq.com
# EU site:            https://api.datadoghq.eu
BACKEND_URL=https://api.datadoghq.com
BACKEND_API_KEY=your_api_key_here
BACKEND_APP_KEY=your_application_key_here
```

Datadog requires **both** an API key and an Application key — a single key is not enough for span/trace queries, even though ingestion only needs the API key. Trace search uses [Datadog's span search query syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/) rather than TraceQL or Jaeger-style tag params, and traces are reconstructed from grouped spans since Datadog has no trace-level lookup endpoint. The backend also refuses a plain `http://` URL — see [What's Different From Upstream](#whats-different-from-upstream).

> **Troubleshooting:** a `403` from the Datadog API almost always means the Application key (not the API key) is missing or invalid. If you're on the EU site, double check `BACKEND_URL` is `https://api.datadoghq.eu`, not the US default.

</details>

<details>
<summary><b>Sentry</b></summary>

```bash
BACKEND_TYPE=sentry
# SaaS (may be region-specific, e.g. https://us.sentry.io):
BACKEND_URL=https://sentry.io
BACKEND_API_KEY=your_auth_token_here
BACKEND_SENTRY_ORG=your-org-slug
# Optional: narrow queries to one project
BACKEND_SENTRY_PROJECT=your-project-slug
```

Sentry requires **both** an auth token *and* an organization slug — every endpoint this backend calls is organization-scoped. Trace search uses [Sentry's search syntax](https://docs.sentry.io/concepts/search/) against the Discover/Explore Events API. Unlike Datadog, Sentry does have a native trace-lookup endpoint, so `get_trace` calls it directly instead of reconstructing a trace from spans — `search_traces` still discovers candidate trace IDs via a span search first, since Sentry's search surface is itself span-centric. Like Datadog, this backend refuses a plain `http://` URL.

> **Troubleshooting:** a `403`/`401` from the Sentry API almost always means the auth token is missing, invalid, or lacks the necessary scopes. A `404` on an org-scoped endpoint usually means the organization slug is wrong. For a self-hosted install, `BACKEND_URL` should be the install's own base URL, not `https://sentry.io`. Some of the tracing endpoints this backend depends on are newer/experimental on Sentry's side and may not be available on every plan or self-hosted version — see the module docstring in [backends/sentry.py](src/opentelemetry_mcp/backends/sentry.py) for specifics.

</details>

<details>
<summary><b>AWS X-Ray</b></summary>

```bash
BACKEND_TYPE=xray
# Decorative only - never dereferenced for a live request. Keep it
# consistent with BACKEND_AWS_REGION by convention.
BACKEND_URL=https://xray.us-east-1.amazonaws.com
BACKEND_AWS_REGION=us-east-1
```

Unlike every other backend here, X-Ray is queried via `boto3`/SigV4, not a bearer token — auth comes from boto3's standard credential chain (environment variables, `~/.aws/credentials`, an assumed role, or an instance/task role). The running process needs `xray:GetTraceSummaries`, `xray:BatchGetTraces`, and (optionally, for faster `list_services`) `xray:GetServiceGraph` IAM permissions.

**Filtering is more limited than the other backends.** By default, OpenTelemetry span attributes are converted to X-Ray segment *metadata*, not *annotations* — only annotations are indexed and queryable via X-Ray's `FilterExpression` search syntax. This backend can only natively push down service name, duration, and error/fault/throttle-derived status; every other field (including all `gen_ai.*` attributes) is always applied client-side after hydrating full traces, unless your own OTel/ADOT collector config explicitly promotes those keys to indexed annotations.

> **Troubleshooting:** an `AccessDeniedException` from `GetServiceGraph` is non-fatal — `list_services` automatically falls back to sampling recent traces. An `AccessDeniedException` from `GetTraceSummaries`/`BatchGetTraces` is fatal for search/hydration and will surface as an unhealthy `health_check`/`doctor` result; double check the IAM permissions above.

</details>

<details>
<summary><b>New Relic</b></summary>

```bash
BACKEND_TYPE=newrelic
# US datacenter (default): https://api.newrelic.com/graphql
# EU datacenter:            https://api.eu.newrelic.com/graphql
BACKEND_URL=https://api.newrelic.com/graphql
BACKEND_API_KEY=your_user_api_key_here
BACKEND_NEWRELIC_ACCOUNT_ID=1234567
```

New Relic requires a **User** API key (not an Ingest or License key, which cannot query) *and* an account ID — a User key is user-scoped, not account-scoped, so every NerdGraph query here states which account to query explicitly. Search uses NRQL (`SELECT * FROM Span WHERE ...`); `get_trace` uses NerdGraph's native `distributedTracing.trace` lookup, like Sentry, rather than reconstructing a trace from search results.

> **Built without a live New Relic account to verify against.** Two things a real account is needed to confirm: the exact shape of OTel span attributes inside `distributedTracing.trace`'s GraphQL response (assumed to be a flat, directly-keyed object), and the account's actual Span-data retention window (a conservative 7-day default is used pending confirmation). See the module docstring in [backends/newrelic.py](src/opentelemetry_mcp/backends/newrelic.py) for the complete list of unverified assumptions before relying on this backend in production.

</details>

<details>
<summary><b>Honeycomb</b></summary>

```bash
BACKEND_TYPE=honeycomb
# US instance (default): https://api.honeycomb.io
# EU instance:            https://api.eu1.honeycomb.io
BACKEND_URL=https://api.honeycomb.io
BACKEND_API_KEY=your_configuration_key_here
BACKEND_HONEYCOMB_DATASET=your-dataset-slug
```

Honeycomb requires a **Configuration Key** (not an Ingest or Management key) with "Manage Queries and Columns" and "Run Queries" permissions, *and* a dataset slug — the Query Data API is dataset-scoped. Search runs Honeycomb's 3-step async query flow (create a query spec, run it, poll for the result) rather than a text query language like NRQL/TraceQL; `get_trace` searches by `trace.trace_id` since no dedicated single-trace lookup endpoint is documented.

> **Go/no-go risk, not just an unverified detail.** The research behind this backend found the programmatic Query Data API — create-query-result + get-query-result, the *only* way to actually run a query and get data back — is documented as **Enterprise-plan exclusive**; Free and Pro accounts can build a query specification via the API but never execute it. This backend raises a clear `HoneycombEnterpriseRequiredError` if a 402/403 is returned, but the exact status code a non-Enterprise account gets back was never confirmed live, and this backend was built entirely without a live account (Enterprise or otherwise). See the module docstring in [backends/honeycomb.py](src/opentelemetry_mcp/backends/honeycomb.py) for the complete list of unverified assumptions - including how raw per-span rows are extracted from Honeycomb's aggregation-oriented Query API - before relying on this backend in production.

</details>

### Transport Modes

```bash
# stdio (default) — local use, Claude Desktop, single process
uvx tracehub-mcp
tracehub-mcp                      # pipx/pip install
uv run tracehub-mcp               # from-source install

# HTTP — remote access, multiple clients, network deployment, sample applications
uvx tracehub-mcp --transport http --host 0.0.0.0 --port 8000
tracehub-mcp --transport http --host 0.0.0.0 --port 8000              # pipx/pip install
uv run tracehub-mcp --transport http --host 0.0.0.0 --port 8000       # from-source install
```

With HTTP transport, clients connect to `http://<host>:<port>/mcp` (streamable-HTTP, for compatibility across MCP clients).

> **Fly.io:** the one-click `fly mcp launch` command only supports stdio-transport servers — it
> can't deploy this server's `--transport http` mode. Running tracehub-mcp on Fly.io with HTTP
> transport needs the manual `fly.toml` + `fly deploy` path instead.

---

## Diagnostics

Validate a config before wiring it into a real MCP client:

```bash
tracehub-mcp doctor --backend jaeger --url http://localhost:16686
```

```
[OK] Configuration loaded and validated
[OK] Backend constructed: jaeger @ http://localhost:16686/
[OK] Health check: healthy
[OK] Connectivity probe (list_services): 2 service(s)
```

`doctor` accepts the same `--backend`/`--url`/etc. flags as the main command and exits non-zero if any step fails - unlike normal server startup, which lazily initializes the backend and deliberately keeps running even if the initial health check fails.

To see the fully-resolved configuration (env vars + CLI overrides merged) without starting the server:

```bash
tracehub-mcp --print-config --backend jaeger --url http://localhost:16686
```

Secrets (`--api-key`/`--app-key`) are reported as `api_key_set`/`app_key_set` booleans, never their actual value.

---

## Security Considerations

Trace and span data returned by this server — attribute values, error messages, operation names — comes from whatever application your observability backend is instrumenting, not from tracehub-mcp itself. That makes it fundamentally the same category of untrusted external content as a webpage or a file, even though it's a trusted server (this one) handing it back to your MCP client.

- **Treat backend data as untrusted input.** An LLM client consuming trace/span data from tracehub-mcp should apply the same caution it would to any other external tool output — a span attribute or error message is application data to reason about, not an instruction to follow, no matter how it's phrased.
- **This is a known MCP risk category, not a tracehub-mcp-specific one.** OWASP's GenAI Security Project covers it in their [Practical Guide for Secure MCP Server Development](https://genai.owasp.org/resource/a-practical-guide-for-secure-mcp-server-development/), and Anthropic's own engineering guidance, [How We Contain Claude](https://www.anthropic.com/engineering/how-we-contain-claude), states plainly that tool output is an attack surface even when the tool itself is trusted.
- **Practical implication:** if you're querying traces from an application that processes untrusted user input (e.g. a customer-facing chatbot), be aware that adversarial content a user fed into that application could end up in a span attribute this server returns — and from there, in your LLM client's context.

---

## Privacy Policy

tracehub-mcp is self-hosted software with no maintainer-operated service or account — it
collects no data of its own. See [PRIVACY.md](PRIVACY.md) for the full policy: what data
the software touches, where network calls go, and how secrets and self-instrumentation
are handled.

---

## MCP Client Setup

Every example below uses `uvx tracehub-mcp` (no install step). Swap in `tracehub-mcp` (pip/pipx install) or `uv run tracehub-mcp` (from-source, `--directory /absolute/path/to/tracehub-mcp`) if you installed it a different way — see [Installation](#installation).

<details>
<summary><b>Claude Desktop</b></summary>

Config file location:
- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

**Jaeger** (no auth):

```json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}
```

**Datadog** (API key + App key):

```json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "datadog",
        "BACKEND_URL": "https://api.datadoghq.com",
        "BACKEND_API_KEY": "your_api_key_here",
        "BACKEND_APP_KEY": "your_application_key_here"
      }
    }
  }
}
```

**Sentry** (auth token + org slug):

```json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "sentry",
        "BACKEND_URL": "https://sentry.io",
        "BACKEND_API_KEY": "your_auth_token_here",
        "BACKEND_SENTRY_ORG": "your-org-slug"
      }
    }
  }
}
```

If you're running from a clone instead, the bundled wrapper script gives easy backend switching during local dev:

```json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "/absolute/path/to/tracehub-mcp/start_locally.sh"
    }
  }
}
```

(the script ships Jaeger/Traceloop/Tempo blocks only, with Jaeger active by default — to switch, comment out the active block and uncomment the one you want; Datadog/Sentry aren't in the script, so add their `export` lines manually).

</details>

<details>
<summary><b>Claude Code</b></summary>

```bash
claude mcp add tracehub-mcp -e BACKEND_TYPE=jaeger -e BACKEND_URL=http://localhost:16686 -- uvx tracehub-mcp
```

Datadog/Sentry work the same way — add more `-e KEY=value` flags for each backend's required env vars (see [Backend-Specific Setup](#backend-specific-setup)). Then:

```bash
claude mcp list
claude "Show me traces with errors from the last hour"
```

</details>

<details>
<summary><b>Cursor</b></summary>

`.cursor/mcp.json` (project-level) or your global Cursor MCP config:

```json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}
```

</details>

<details>
<summary><b>Windsurf</b></summary>

`~/.codeium/windsurf/mcp_config.json`:

```json
{
  "mcpServers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}
```

</details>

<details>
<summary><b>VS Code (GitHub Copilot)</b></summary>

`.vscode/mcp.json` in your workspace — note the top-level key is `servers`, not `mcpServers`, and stdio servers need no `"type"` field:

```json
{
  "servers": {
    "tracehub-mcp": {
      "command": "uvx",
      "args": ["tracehub-mcp"],
      "env": {
        "BACKEND_TYPE": "jaeger",
        "BACKEND_URL": "http://localhost:16686"
      }
    }
  }
}
```

</details>

<details>
<summary><b>Gemini CLI</b></summary>

Config file: `~/.gemini/config.json`, same JSON shape as Claude Desktop above. Then:

```bash
gemini "Analyze token usage for gpt-4 requests today"
```

</details>

---

## Tools Reference

tracehub-mcp exposes **19 MCP tools**:

| Tool                       | Description                                       | Use Case                           |
| --------------------------- | -------------------------------------------------- | ----------------------------------- |
| `search_traces`            | Search traces with simple params or advanced filters | Find specific requests or patterns |
| `search_spans_tool`        | Search individual spans (not grouped into traces) | Find LLM tool calls, specific ops  |
| `get_trace`                | Get complete trace details by trace ID            | Deep-dive into a single trace      |
| `triage_trace`             | Synthesize a likely-root-cause diagnosis for a trace (critical path, latency ranking, error chain) | Act on a diagnosis instead of re-deriving one from a trace dump |
| `correlate_trace`          | Find the corresponding trace in a second, independently-configured backend | Join a trace across two backends (e.g. Datadog → Sentry) |
| `get_llm_usage`             | Aggregate token usage metrics                     | Track costs and usage trends       |
| `list_services`            | List available services                           | Discover what's instrumented       |
| `find_errors`               | Find traces with errors                           | Debug failures quickly             |
| `list_llm_models`          | Discover models in use, with usage stats          | Track model adoption, shadow AI    |
| `get_llm_model_stats`      | Latency/token percentiles + finish reasons for one model | Compare model efficiency    |
| `get_llm_expensive_traces` | Find highest token-usage traces                   | Cost optimization                  |
| `get_llm_slow_traces`      | Find slowest traces by duration                   | Latency debugging                  |
| `list_llm_tools_tool`      | Discover LLM tool/function calls (`traceloop.span.kind == tool`) | Track agent tool usage |
| `list_sessions`            | Group spans by `gen_ai.conversation.id`           | Understand multi-turn conversation activity |
| `get_session_stats`        | Detailed stats for one conversation ID            | Drill into a single conversation   |
| `compare_time_windows`     | Diff aggregated usage between two time ranges     | "This week vs last week" comparisons |
| `investigate_cost_spike`   | Compare cost between a recent window and a baseline, ranked by model/service | "Why did our LLM bill spike?" |
| `investigate_error_spike`  | Compare error rate between a recent window and a baseline, ranked by service/model/error type | "Is this error increase a real spike?" |
| `get_prompt_version_stats` | Group spans by `gen_ai.prompt.name`/`.version`    | Compare prompt versions before promoting one |

### Backend Support Matrix

| Feature          | Jaeger | Tempo | Traceloop | Datadog | Sentry | AWS X-Ray | New Relic | Honeycomb♦ |
| ---------------- | :----: | :---: | :-------: | :-----: | :----: | :-------: | :-------: | :--------: |
| Search traces    |   ✓    |   ✓   |     ✓     |   ✓†    |   ✓‡   |    ✓†     |    ✓†     |    ✓†      |
| Search spans     |  ✓\*   |   ✓   |     ✓     |    ✓    |   ✓    |    ✓†     |    ✓      |    ✓       |
| Get trace by ID  |   ✓    |   ✓   |     ✓     |   ✓†    |   ✓    |    ✓      |    ✓‡     |    ✓†      |
| Advanced filters |   ✓    |   ✓   |     ✓     |    ✓    |   ✓    |    ✓§     |    ✓      |    ✓       |
| Error traces     |   ✓    |   ✓   |     ✓     |    ✓    |   ✓    |    ✓      |    ✓      |    ✓       |
| All LLM tools    |   ✓    |   ✓   |     ✓     |    ✓    |   ✓    |    ✓      |   ✓¶      |   ✓¶       |

<sub>\* Jaeger requires the `service_name` parameter for span search.</sub><br>
<sub>† Datadog, AWS X-Ray, New Relic, and Honeycomb have no trace-level *search* API (X-Ray's `BatchGetTraces` fetches by exact ID); traces are reconstructed by searching (spans, X-Ray trace summaries, NRQL Span events, or Honeycomb breakdown groups) and hydrating, with every result re-verified against the exact ID requested.</sub><br>
<sub>‡ Sentry and New Relic both have a native trace-lookup endpoint (unlike Datadog and Honeycomb), so `get_trace` calls it directly; `search_traces` still discovers candidate trace IDs via a span/NRQL search first, since both backends' search surface is itself span-centric.</sub><br>
<sub>§ AWS X-Ray natively filters only service name, duration, and error/fault/throttle-derived status — OTel attributes (including all `gen_ai.*` fields) land in unindexed segment *metadata* by default, not indexed *annotations*, so they're always applied client-side rather than pushed down to `FilterExpression`.</sub><br>
<sub>¶ New Relic: `search_spans` reads `gen_ai.*` attributes directly off NRQL's well-documented flat row shape; `search_traces`/`get_trace` additionally rely on an unverified assumption about the shape of the `attributes` field inside NerdGraph's `distributedTracing.trace` response, mitigated by overlaying the original search row's attributes back onto the hydrated result. Honeycomb: `gen_ai.*` attributes are extracted via the same breakdown-column technique used for every other field (module docstring point 2), an equally unverified extraction method.</sub><br>
<sub>♦ Honeycomb's entire "Search"/"Get trace by ID" row of checkmarks above depends on the Query Data API, which the research behind this backend found is documented as **Enterprise-plan exclusive** - a Free/Pro account may not be able to exercise any of this at all. See the Honeycomb setup section above.</sub>

### Key Tool Details

**`search_traces`**

```python
{
    "service_name": "my-app",
    "start_time": "2024-01-01T00:00:00Z",
    "end_time": "2024-01-01T23:59:59Z",
    "gen_ai_system": "openai",
    "gen_ai_request_model": "gpt-4",
    "min_duration_ms": 1000,
    "has_error": false,
    "limit": 50,
}
```

Parameters: `service_name`, `operation_name`, `start_time`/`end_time` (ISO 8601), `min_duration_ms`/`max_duration_ms`, `gen_ai_system`, `gen_ai_request_model`, `gen_ai_response_model`, `has_error`, `tags`, `filters` (see [Generic Filter System](#generic-filter-system)), `limit` (1-1000, default 100). Returns trace summaries with token counts.

**`get_trace`**

```python
{"trace_id": "abc123def456"}
```

Returns the full trace tree: all spans with attributes, parsed OpenTelemetry `gen_ai.*` data for LLM spans, per-span token usage, error information, and each span's raw `events` (e.g. `gen_ai.evaluation.result`, or any other instrumentation-emitted event — not filtered to a fixed set of names).

**`triage_trace`**

```python
{"trace_id": "abc123def456", "detail_level": "summary"}
```

Instead of returning raw trace data for an agent to re-derive a diagnosis from every time, this synthesizes one directly: a critical path (the "Last Finishing Child" chain actually responsible for the trace's total latency), the top spans ranked by self-time (latency contribution net of children), and — when the trace contains an error — the deepest error span in the trace's error chain as the likely root cause (`confidence: "high"`, or `"medium"` when multiple equally-deep error chains make blame ambiguous). Falls back to the highest self-time span as a pure-latency diagnosis (`confidence: "low"`) when no error is present. Deterministic — no LLM call — and works against any configured backend, since it operates on `get_trace`'s already-fetched span data. `detail_level: "full"` additionally attaches the diagnosed root cause's raw error detail (message/type/stacktrace) when the verdict is error-driven.

**`correlate_trace`**

```python
{"trace_id": "abc123def456"}
```

Tries to find the corresponding trace in a second, independently-configured backend (e.g. a Datadog trace and its downstream Sentry error, joined). Tries a direct `trace_id` match in the secondary backend first (`confidence: "high"`); if that fails, falls back to a time-window + service-name-overlap heuristic search (`confidence: "low"`) - queried once per service name in the primary trace, so this works against backends like Jaeger that require `service_name` on `search_traces`. Each match also reports `root_cause_consistent`: `null` when neither trace has an error to compare, otherwise `true`/`false` for whether both sides agree on the error and which service it traces back to (reusing the same error-chain logic `triage_trace` uses). This is a best-effort correlation, not a guaranteed join - the result always includes a fixed `limitations` list (clock skew, sampling mismatches, partial trace visibility, and the fact that trace-id continuity across a vendor boundary isn't guaranteed even under normal W3C Trace Context propagation). Requires a secondary backend configured via `SECONDARY_BACKEND_TYPE`/`SECONDARY_BACKEND_URL` (see [configuration table](#all-configuration-options)); raises a clear error otherwise.

**`get_llm_usage`**

```python
{
    "start_time": "2024-01-01T00:00:00Z",
    "end_time": "2024-01-01T23:59:59Z",
    "service_name": "my-app",
    "gen_ai_system": "openai",
    "limit": 1000,
}
```

Returns aggregated prompt/completion/total tokens, broken down by model and by service, plus request counts.

**`list_services`** — no parameters. Returns the list of instrumented service names.

**`find_errors`**

```python
{"start_time": "2024-01-15T14:00:00Z", "service_name": "my-app", "limit": 50}
```

Returns error messages, error types, truncated stack traces, and LLM-specific error info.

**`list_llm_models` / `get_llm_model_stats` / `get_llm_expensive_traces` / `get_llm_slow_traces` / `list_llm_tools_tool` / `search_spans_tool` / `list_sessions` / `get_session_stats` / `compare_time_windows` / `investigate_cost_spike` / `investigate_error_spike` / `get_prompt_version_stats`** are documented in detail, with worked examples, in [CLAUDE.md](CLAUDE.md) — this README covers the shape every tool shares; CLAUDE.md is the fuller reference for exact parameters and response fields on the LLM-analysis tools.

---

## Generic Filter System

`search_traces` and `search_spans_tool` both accept a `filters` list in addition to (or instead of) their simple named parameters, for advanced queries. Each filter is:

```json
{
  "field": "gen_ai.usage.total_tokens",
  "operator": "gt",
  "value": 5000,
  "value_type": "number"
}
```

- **`field`** — dotted attribute name, e.g. `gen_ai.usage.prompt_tokens`, `traceloop.span.kind`, `service.name`
- **`operator`** — see table below
- **`value`** — single value (most operators) or **`values`** — list (for `in`, `not_in`, `between`)
- **`value_type`** — `"string"`, `"number"`, or `"boolean"`

| Category    | Operators                                                            |
| ----------- | --------------------------------------------------------------------- |
| String      | `equals`, `not_equals`, `contains`, `not_contains`, `starts_with`, `ends_with`, `in`, `not_in` |
| Number      | `equals`, `not_equals`, `gt`, `lt`, `gte`, `lte`, `between`, `in`, `not_in` |
| Boolean     | `equals`, `not_equals`                                                |
| Existence   | `exists`, `not_exists` (no value needed)                              |

Multiple filters combine with **AND** logic. Legacy simple parameters (`service_name`, `gen_ai_request_model`, etc.) still work and are converted to filters internally — mix and match freely.

The server uses a **hybrid filtering strategy**: filters are pushed to the backend's native query language when supported (TraceQL for Tempo, span-search syntax for Datadog, Discover syntax for Sentry), and applied client-side afterward for anything the backend can't express natively.

| Backend             | Native filter support                                                | Notes                          |
| -------------------- | ---------------------------------------------------------------------- | ------------------------------- |
| **Tempo** (TraceQL)  | `equals`, `not_equals`, `gt`, `lt`, `gte`, `lte`, `contains` (regex), `in` (OR), `exists`, `not_exists` | — |
| **Traceloop**        | `equals`, `not_equals`, `gt`, `lt`, `gte`, `lte`                       | —                                |
| **Datadog**          | Most operators via span-search syntax                                 | Field names validated against an allowlist before being spliced into the query |
| **Sentry**           | Most operators via Discover search syntax                             | Same field-name allowlisting as Datadog |
| **Jaeger**           | `equals` (via tags only)                                               | **Requires `service_name`**    |

Example — expensive OpenAI traces:

```json
{
  "filters": [
    { "field": "gen_ai.system", "operator": "equals", "value": "openai", "value_type": "string" },
    { "field": "gen_ai.usage.total_tokens", "operator": "gt", "value": 5000, "value_type": "number" }
  ]
}
```

For the full semantic-convention attribute list (`gen_ai.*` vs legacy `llm.*`, token-naming variants across providers, finish-reason values, and the token-calculation fallback chain), see [CLAUDE.md](CLAUDE.md#opentelemetry-semantic-conventions).

---

## Example Queries

### Find Expensive OpenAI Operations

**Ask:** _"Show me OpenAI traces from the last hour that took longer than 5 seconds"_

**Tool call:** `search_traces`

```json
{
  "service_name": "my-app",
  "gen_ai_system": "openai",
  "min_duration_ms": 5000,
  "start_time": "2024-01-15T10:00:00Z",
  "limit": 20
}
```

**Response:**

```json
{
  "traces": [
    {
      "trace_id": "abc123...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "status": "OK",
      "duration_ms": 8250,
      "span_count": 3,
      "llm_span_count": 1,
      "total_tokens": 4523,
      "has_errors": false
    }
  ],
  "count": 1
}
```

### Analyze Token Usage by Model

**Ask:** _"How many tokens did we use for each model today?"_

**Tool call:** `get_llm_usage`

```json
{
  "start_time": "2024-01-15T00:00:00Z",
  "end_time": "2024-01-15T23:59:59Z",
  "service_name": "my-app"
}
```

**Response:**

```json
{
  "period": { "start_time": "2024-01-15T00:00:00Z", "end_time": "2024-01-15T23:59:59Z" },
  "filters": { "service_name": "my-app" },
  "summary": {
    "total_requests": 487,
    "total_prompt_tokens": 82140,
    "total_completion_tokens": 43290,
    "total_tokens": 125430
  },
  "by_model": {
    "gpt-4": { "requests": 156, "prompt_tokens": 58300, "completion_tokens": 26900, "total_tokens": 85200 },
    "gpt-3.5-turbo": { "requests": 331, "prompt_tokens": 23840, "completion_tokens": 16390, "total_tokens": 40230 }
  },
  "by_service": {
    "my-app": { "requests": 487, "prompt_tokens": 82140, "completion_tokens": 43290, "total_tokens": 125430 }
  }
}
```

### Find Traces with Errors

**Ask:** _"Show me all errors from the last hour"_

**Tool call:** `find_errors`

```json
{
  "start_time": "2024-01-15T14:00:00Z",
  "service_name": "my-app",
  "limit": 10
}
```

**Response:**

```json
{
  "count": 1,
  "error_traces": [
    {
      "trace_id": "def456...",
      "service_name": "my-app",
      "operation_name": "chat.completions",
      "start_time": "2024-01-15T14:23:15Z",
      "duration_ms": 1200,
      "status": "ERROR",
      "span_count": 2,
      "llm_span_count": 1,
      "total_tokens": 310,
      "has_errors": true,
      "error_spans": [
        {
          "span_id": "span789...",
          "operation_name": "chat.completions",
          "service_name": "my-app",
          "status": "ERROR",
          "error_message": "RateLimitError: Too many requests",
          "error_type": "openai.error.RateLimitError",
          "is_llm_error": true,
          "llm_provider": "openai",
          "llm_model": "gpt-4"
        }
      ]
    }
  ]
}
```

### Compare Model Performance

**Ask:** _"What's the performance difference between GPT-4 and Claude?"_

**Tool call 1:** `get_llm_model_stats` for `gpt-4`

```json
{ "model_name": "gpt-4", "start_time": "2024-01-15T00:00:00Z" }
```

**Tool call 2:** `get_llm_model_stats` for `claude-3-opus`

```json
{ "model_name": "claude-3-opus-20240229", "start_time": "2024-01-15T00:00:00Z" }
```

### Investigate High Token Usage

**Ask:** _"Which requests used the most tokens today?"_

**Tool call:** `get_llm_expensive_traces`

```json
{ "limit": 10, "start_time": "2024-01-15T00:00:00Z", "min_tokens": 5000 }
```

---

## Common Workflows

### Cost Optimization

1. `get_llm_expensive_traces` — find the highest-token requests
2. `get_llm_usage` — see which models are costing the most
3. `get_trace` on a specific `trace_id` — inspect the exact prompt/response

### Performance Debugging

1. `get_llm_slow_traces` — identify latency outliers
2. `find_errors` — check for failure patterns
3. `get_llm_model_stats` — check finish-reason distribution for truncation

### Model Adoption Tracking

1. `list_llm_models` — see every model actually being called
2. `get_llm_model_stats` per model — compare performance
3. Scan `list_llm_models` results for unexpected models/services (shadow AI)

---

## Development

```bash
git clone https://github.com/mcpsmiths/tracehub-mcp.git
cd tracehub-mcp
uv sync --group dev   # pulls in pytest, mypy, ruff, etc. for local iteration

# Tests (458 passed, 2 skipped at time of writing)
uv run pytest

# With coverage
uv run pytest --cov=opentelemetry_mcp --cov-report=html

# Format, lint, type-check
uv run ruff format .
uv run ruff check .
uv run mypy src/
```

CI (`.github/workflows/ci.yml`) runs Ruff and mypy (strict) on every push, plus the full pytest suite.

---

## Troubleshooting

**Backend connection issues:**

```bash
curl http://localhost:16686/api/services   # Jaeger
curl http://localhost:3200/api/search/tags  # Tempo
```

**Authentication errors:** confirm your key is set —

```bash
export BACKEND_API_KEY=your_key_here
# or: tracehub-mcp --api-key your_key_here
```

**No traces found:**
- Check the time range (use recent timestamps)
- Verify service names with `list_services`
- Try searching without filters first

**Token usage shows zero:**
- Confirm your traces have OpenTelemetry `gen_ai.*` (or legacy `llm.*`) instrumentation
- Inspect raw span attributes with `get_trace`

**Datadog/Sentry-specific issues:** see the troubleshooting notes under each backend in [Configuration](#configuration).

---

## Roadmap

Both ideas that were previously listed here as deferred are now **shipped**:

- Agent-native triage — tools that flag a likely root cause rather than just returning raw trace data: see `triage_trace` in the [Tools Reference](#tools-reference) above.
- Cross-backend correlation — querying a second, independently-configured backend and correlating results against it: see `correlate_trace` above. This ships as a best-effort, heuristic correlation (direct `trace_id` match, or a time-window/service-overlap fallback) rather than a guaranteed schema-level join, per the research behind it — trace-id continuity across a vendor boundary is not guaranteed even under normal W3C Trace Context propagation.

New Relic and Honeycomb have since **shipped** as the 7th and 8th backends, both built entirely from research/documentation without a live account to verify against — see their own [setup sections](#backend-specific-setup) for exactly what's unverified before relying on either in production; Honeycomb in particular carries a real go/no-go risk (its programmatic Query Data API is documented as Enterprise-plan exclusive). There is no further backend currently under research. (Grafana Cloud was never on this list — it already ships today via Tempo's Basic Auth path; see [Grafana Tempo](#backend-specific-setup) above.)

Carried over from upstream's older roadmap, re-prioritized behind the above rather than dropped: a dedicated model-vs-model comparison tool (today's `get_llm_model_stats` and `compare_time_windows` cover per-model stats and time-window diffing separately, but not a single tool that diffs two specific models directly), broader prompt-pattern analysis across templates rather than just version-over-version for one prompt (`get_prompt_version_stats` already covers the latter), MCP resources for common queries, and SigNoz/ClickHouse backend support. Cost calculation with built-in pricing tables and a query-result caching layer are **shipped**, not pending: every usage-reporting tool (e.g. `investigate_cost_spike` in the [Tools Reference](#tools-reference)) returns a `cost_usd`/`cost_usd_is_partial` pair backed by a vendored litellm pricing table (`src/opentelemetry_mcp/pricing/`), and query results are cached with in-flight request coalescing via the `QUERY_CACHE_TTL_SECONDS` env var (see the [configuration table](#all-configuration-options) above).

Everything else documented elsewhere in this README is shipped.

---

## Contributing

Contributions are welcome. Before opening a PR, make sure:

1. All tests pass: `uv run pytest`
2. Code is formatted: `uv run ruff format .`
3. No linting errors: `uv run ruff check .`
4. Type checking passes: `uv run mypy src/`

## License

Apache License 2.0 — see [LICENSE](LICENSE). This project is a fork of [traceloop/opentelemetry-mcp-server](https://github.com/traceloop/opentelemetry-mcp-server); full attribution and the fork relationship are documented in [NOTICE](NOTICE).

## Support

- **Issues & feature requests:** [github.com/mcpsmiths/tracehub-mcp/issues](https://github.com/mcpsmiths/tracehub-mcp/issues)
- **Changelog:** [CHANGELOG.md](CHANGELOG.md)
- **Community:** [Traceloop Community Slack](https://traceloop.com/slack) (upstream's community channel; general OpenTelemetry/GenAI-tracing questions welcome)
- **Related:** [Model Context Protocol](https://modelcontextprotocol.io/) · [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) · [traceloop/opentelemetry-mcp-server](https://github.com/traceloop/opentelemetry-mcp-server) (upstream)