Skip to main content
Glama

otel-mcp-server

CI Live Docker Tests License: Apache-2.0

An MCP server that exposes your OpenTelemetry observability stack — traces, metrics, logs, and more — as tools for AI agents. Built on a Skill plugin architecture for easy extensibility.

Give any LLM agent the ability to query your Jaeger traces, run PromQL, search Loki logs, and investigate production issues — through a standard protocol.

┌─────────────────┐      MCP     ┌──────────────────┐──► Traces (Jaeger · Zipkin · Tempo · SkyWalking)
│  Claude Desktop │ ◄──────────► │                  │──► Prometheus · InfluxDB · OpenTSDB
│  GitHub Copilot │ (stdio/HTTP) │  otel-mcp-server │──► Loki · ClickHouse · Graylog (logs)
│  Custom Agent   │              │                  │──► Pinpoint · Elasticsearch · Alertmanager
└─────────────────┘              │   26 skills      │──► Grafana · Pyroscope · OPA
                                 │   114 tools      │──► Cilium · Kubernetes (eBPF/CRDs)
                                 │   authenticated  │──► Envoy · Consul · Kong · Traefik
                                 └──────────────────┘──► Fluent Bit · Beats · Vector · Alloy
                                                     └─► App API    (ZK/system)

Example

"What's running, what's healthy, and what needs attention?" — answered in seconds by an AI agent using this MCP server against a local Docker Compose stack:

MCP server exploring a local observability stack — showing Jaeger, Prometheus, and Loki backend status, active alerts, and key findings

"Tell me what happened to order ORD-1774382223417-7" — full distributed trace across 4 services, 40 spans, with ZK proof verification:

Order tracing across gateway, exchange, matcher, and wallet services — showing timeline, fill price, latency breakdown, and Groth16 ZK proof verification

"What about the k8s cluster?"

Health check across the cluster

Related MCP server: local-devops-mcp

Features

  • 114 tools across 26 skills (+6 opt-in write tools) — a provider-agnostic traces layer (Jaeger/Zipkin/Tempo/SkyWalking via TRACES_PROVIDER), metrics (Prometheus/InfluxDB/OpenTSDB), logs (Loki/ClickHouse/Graylog), Pinpoint, Elasticsearch, Alertmanager, vmalert rule evaluation, Grafana, Cilium, Grafana Beyla (eBPF auto-instrumentation), Kubernetes, Pyroscope, OPA, service mesh (Envoy/Consul/Kong/Traefik), collection pipelines (Fluent Bit/Beats/Vector/Alloy), AgentRelay agent coordination, ZK proofs, system health, public exchange transparency

  • Skill plugin architecture — each backend is a self-contained plugin; add new ones with a single file

  • Two transports — stdio (Claude Desktop, Copilot) and HTTP (remote, multi-client)

  • MCP 2026-07-28 ready — serves the stateless protocol core (no handshake, no session id) alongside ≤2025-11-25 clients on the same endpoint, validates the Mcp-Method/Mcp-Name routing headers, emits ttlMs/cacheScope cache hints, answers server/discover, and propagates W3C trace context from _meta into backend queries (see MCP 2026-07-28 support)

  • Layered auth — backend credentials (Bearer/Basic/OAuth client-credentials/custom headers per backend) and client identity via static API keys, Just-in-Time scoped ephemeral tokens (OWASP MCP Top 10), or enterprise-managed authorization through your corporate IdP (MCP ext-auth ID-JAG). allowedTools scopes are enforced per session (see Client Authentication)

  • Selective skills — enable only the skills you need (--tools traces,metrics,logs)

  • Multi-version aware — a typed capability → product → protocol-adapter model tracks which versions and protocol features each backend supports; runtime detection surfaces live product/version on /health, and MCP_VERSION_GATING (off/warn/enforce) can guard version-sensitive features (unknown versions always pass optimistically)

  • Multi-backend & failover — a single skill can address multiple named instances and fail over across replicas; tools accept an optional SSRF-safe target argument (see Multi-backend instances & failover)

  • Self-metrics — GET /metrics endpoint with tool call counts, backend latencies, auth attempts, and JIT token issuance/rotation/revocation/denial counters

  • Container-native — env-var config, K8s Secret mounting, multi-stage Dockerfile

  • Zero dependencies beyond the MCP SDK and Zod

For role-based Studio workflows, see docs/studio-user-journeys.md.

Quick Start

New here? docs/getting-started.md walks the whole flow end to end — install, point at your backends, wire up Claude Desktop, and the first questions to ask — in about five minutes. The rest of this section is the condensed reference.

Install

Run directly with npx (no clone or build needed):

npx -y @moebiusx/otel-mcp-server

Or install globally:

npm install -g @moebiusx/otel-mcp-server
otel-mcp-server
git clone https://github.com/MoebiusX/otel-mcp-server.git
cd otel-mcp-server
npm install
npm run build

Run (stdio — for Claude Desktop / Copilot)

# Point at your backends
export JAEGER_URL=http://localhost:16686
export PROMETHEUS_URL=http://localhost:9090
export LOKI_URL=http://localhost:3100

node dist/index.js

Run (HTTP — for remote agents / containers)

node dist/index.js --http 3001
# ✓ otel-mcp-server v1.8.0 listening on http://0.0.0.0:3001
#   Health:  http://localhost:3001/health
#   Metrics: http://localhost:3001/metrics
#   Skills:  6/26 configured — 29 tools registered
#     ✓ traces         — Distributed Traces (5 tools) [Jaeger]
#     ✓ metrics        — Prometheus Metrics (6 tools) [Prometheus]
#     ✓ logs           — Structured Logs (4 tools) [Loki]
#     ✗ elasticsearch  — not configured
#     …
#     ✓ zk-proofs      — ZK Proofs (4 tools) [App API]
#     ✓ system         — System Health (5 tools) [App API, Jaeger]
#     ✓ public-exchange — Public Exchange (5 tools) [App API]

The same inventory is printed on stdio (to stderr, where MCP clients log it), so a missing backend is visible in either transport.

Docker

Published multi-arch images (amd64/arm64) — the entrypoint defaults to --http 3001:

docker run --rm -p 3001:3001 \
  -e PROMETHEUS_URL=http://host.docker.internal:9090 \
  moebiusx/otel-mcp-server:latest

localhost inside the container is the container — use host.docker.internal (Docker Desktop) or a compose/K8s service name to reach backends on the host or in the cluster.

docker build -t otel-mcp-server .
docker run -p 3001:3001 \
  -e JAEGER_URL=http://jaeger:16686 \
  -e PROMETHEUS_URL=http://prometheus:9090 \
  -e LOKI_URL=http://loki:3100 \
  -e ELASTICSEARCH_URL=http://elasticsearch:9200 \
  -e ALERTMANAGER_URL=http://alertmanager:9093 \
  -e GRAFANA_URL=http://grafana:3000 \
  -e GRAFANA_AUTH_TOKEN=glsa_xxx \
  -e MCP_AUTH_KEYS='{"keys":[{"id":"agent-1","key":"sk-my-secret-key"}]}' \
  otel-mcp-server

Configuration

All configuration is via environment variables. The commonly used backend, auth, and runtime variables are listed below.

Backend URLs

Variable

Default

Description

TRACES_PROVIDER

jaeger

Trace backend selector — one of jaeger, tempo, zipkin, skywalking

JAEGER_URL / TRACES_JAEGER_URL

http://localhost:16686

Jaeger Query API (used when TRACES_PROVIDER=jaeger)

TEMPO_URL / TRACES_TEMPO_URL

http://localhost:3200

Grafana Tempo (used when TRACES_PROVIDER=tempo)

ZIPKIN_URL / TRACES_ZIPKIN_URL

http://localhost:9411

Zipkin v2 API (used when TRACES_PROVIDER=zipkin)

SKYWALKING_URL / TRACES_SKYWALKING_URL

http://localhost:12800

SkyWalking OAP GraphQL (used when TRACES_PROVIDER=skywalking)

PROMETHEUS_URL

http://localhost:9090

Prometheus API

LOKI_URL

http://localhost:3100

Loki API

PROMETHEUS_PATH_PREFIX

(empty)

Path prefix (e.g. /prometheus)

APP_API_URL

http://localhost:5000

Application API (for ZK/system tools)

ELASTICSEARCH_URL

(disabled)

Elasticsearch / OpenSearch API

ALERTMANAGER_URL

(disabled)

Alertmanager API

VMALERT_URL

(disabled)

vmalert rules + alerts API

GRAFANA_URL

(disabled)

Grafana API

CILIUM_URL

(disabled)

Cilium agent REST API (eBPF networking)

BEYLA_PROMETHEUS_URL

(disabled)

Prometheus store holding Grafana Beyla's eBPF metrics

KUBERNETES_URL

(in-cluster)

kube-apiserver; auto-detected in-cluster via the ServiceAccount mount

CLICKHOUSE_URL

(disabled)

ClickHouse HTTP interface

PYROSCOPE_URL

(disabled)

Pyroscope HTTP API (continuous profiling)

OPA_URL

(disabled)

Open Policy Agent REST API

ENVOY_ADMIN_URL

(disabled)

Envoy admin API

CONSUL_URL

(disabled)

Consul HTTP API

KONG_ADMIN_URL

(disabled)

Kong Admin API

TRAEFIK_URL

(disabled)

Traefik API

INFLUX_URL

(disabled)

InfluxDB HTTP API (InfluxQL)

OPENTSDB_URL

(disabled)

OpenTSDB HTTP API

GRAYLOG_URL

(disabled)

Graylog REST API

PINPOINT_URL

(disabled)

Pinpoint web API

FLUENTBIT_URL

(disabled)

Fluent Bit HTTP monitoring server

BEATS_URL

(disabled)

Beats HTTP monitoring endpoint

VECTOR_URL

(disabled)

Vector API (GraphQL + health)

ALLOY_URL

(disabled)

Grafana Alloy

AGENTRELAY_URL

(disabled)

AgentRelay hosted REST API (agent coordination)

GRAFANA_DEFAULT_FROM

now-1h

Default Grafana query range start

GRAFANA_MAX_ITEMS

50

Default Grafana list/search limit

MCP_ENABLE_WRITES

(off)

Enable mutating/write tools (e.g. Grafana dashboard provisioning). Read-only by default

MCP_TIMEOUT_MS

15000

Backend query timeout (ms)

MCP_SESSION_IDLE_MS

300000

HTTP transport only: idle time before an inactive session is reaped (ms). Bounds the session map for clients that disconnect without sending a DELETE

MCP_SESSION_SWEEP_MS

60000

HTTP transport only: how often the idle-session reaper runs (ms)

MCP_JIT_MODE

off

JIT privileged identity: off | enabled (migration) | required (static keys may only mint tokens). See Just-in-Time (JIT) Privileged Identity

MCP_JIT_TTL_SECONDS

900

JIT token TTL and the cap for a requested ttlSeconds (clamped 60–3600)

MCP_JIT_MAX_LIFETIME_SECONDS

28800

Hard cap per JIT token lineage (clamped ttl–86400)

MCP_JIT_MAX_ACTIVE_TOKENS

1000

Active-token capacity guard

MCP_JIT_STORE

memory

Where JIT token + single-use replay state lives. memory is in-process (a restart invalidates outstanding tokens). Any other value is a module specifier for an operator-supplied shared-store adapter, needed for multi-replica deployments — see High availability

MCP_ENTERPRISE_AUTH_ISSUER

(disabled)

Trusted enterprise IdP issuer URL (https required; loopback http allowed for local IdP simulators). Set with _AUDIENCE to enable Enterprise-Managed Authorization

MCP_ENTERPRISE_AUTH_AUDIENCE

(disabled)

This server's issuer identifier — the ID-JAG aud must contain it

MCP_ENTERPRISE_AUTH_RESOURCE

(= audience)

This server's resource identifier — the ID-JAG resource must match it

MCP_ENTERPRISE_AUTH_JWKS_URL

(OIDC discovery)

Explicit IdP JWKS endpoint (https required); defaults to discovery from the issuer

MCP_ENTERPRISE_AUTH_DEFAULT_SCOPES

(all enabled)

Scopes granted when an ID-JAG carries no scope claim

MCP_ENTERPRISE_AUTH_MAX_REDEEMED_JTIS

50000

Redeemed-jti replay-cache cap (fail-closed at the cap); size above peak assertion-rate × TTL

Backend Authentication

The MCP server authenticates to each backend independently. For each backend prefix (JAEGER_, TEMPO_, ZIPKIN_, SKYWALKING_, PROMETHEUS_, LOKI_, APP_API_, ELASTICSEARCH_, ALERTMANAGER_, GRAFANA_, CILIUM_, CLICKHOUSE_, PYROSCOPE_, OPA_, ENVOY_, CONSUL_, KONG_, TRAEFIK_, INFLUX_, OPENTSDB_, GRAYLOG_, PINPOINT_, FLUENTBIT_, BEATS_, VECTOR_, ALLOY_, AGENTRELAY_, VMALERT_), you can set:

Suffix

Effect

_AUTH_TOKEN

Sets Authorization: Bearer <token>

_AUTH_BASIC

Sets Authorization: Basic <base64(user:pass)> — provide as user:password

_AUTH_HEADER

Sets Authorization: <raw value> (overrides token/basic)

Special:

Variable

Effect

LOKI_TENANT_ID

Sets X-Scope-OrgID header for multi-tenant Loki

GRAFANA_ORG_ID

Sets X-Grafana-Org-Id header for multi-org Grafana

Kubernetes uses its own credential scheme rather than the prefix above: it presents a ServiceAccount bearer token (auto-loaded from the in-cluster mount, or KUBERNETES_TOKEN / KUBERNETES_TOKEN_FILE) and validates TLS against the cluster CA (KUBERNETES_CA_FILE, or the in-cluster mount). See .env.example.

Example — Prometheus behind OAuth proxy + multi-tenant Loki:

PROMETHEUS_AUTH_TOKEN=eyJhbGci...
LOKI_AUTH_TOKEN=my-loki-token
LOKI_TENANT_ID=team-platform

OAuth 2.0 / OIDC (client-credentials)

When no static _AUTH_* var is set for a backend, the server can obtain a bearer token via the OAuth 2.0 client-credentials grant and refresh it transparently (cached in-memory, refreshed ~60s before expiry, concurrent requests de-duped). Client secrets are never logged or echoed in error messages. Use the same <PREFIX> as above with _AUTH_OAUTH_* suffixes:

Suffix

Effect

_AUTH_OAUTH_CLIENT_ID

OAuth client ID (required)

_AUTH_OAUTH_CLIENT_SECRET

OAuth client secret (required)

_AUTH_OAUTH_TOKEN_URL

Explicit token endpoint (skips OIDC discovery)

_AUTH_OAUTH_ISSUER

OIDC issuer — token endpoint is discovered from /.well-known/openid-configuration

_AUTH_OAUTH_SCOPE

Requested scope (optional)

_AUTH_OAUTH_AUDIENCE

Requested audience (optional; Entra derives .default scope from it)

_AUTH_OAUTH_PROVIDER

Preset: entra / azure / azuread, google, or oidc

_AUTH_OAUTH_TENANT

Entra/Azure tenant ID (with the entra preset)

# Generic OIDC (token endpoint auto-discovered from the issuer)
PROMETHEUS_AUTH_OAUTH_ISSUER=https://idp.example.com/realms/obs
PROMETHEUS_AUTH_OAUTH_CLIENT_ID=otel-mcp
PROMETHEUS_AUTH_OAUTH_CLIENT_SECRET=...
PROMETHEUS_AUTH_OAUTH_SCOPE=metrics:read

# Microsoft Entra ID (Azure AD) preset
TEMPO_AUTH_OAUTH_PROVIDER=entra
TEMPO_AUTH_OAUTH_TENANT=00000000-0000-0000-0000-000000000000
TEMPO_AUTH_OAUTH_CLIENT_ID=...
TEMPO_AUTH_OAUTH_CLIENT_SECRET=...
TEMPO_AUTH_OAUTH_AUDIENCE=api://obs-backend       # → scope api://obs-backend/.default

Static _AUTH_TOKEN / _AUTH_BASIC / _AUTH_HEADER always take precedence, so existing configs are unaffected.

Multi-backend instances & failover

A single skill can talk to multiple named backends and fail over across replicas. The single-URL config above keeps working unchanged — it simply becomes the default instance. Supported skills today: metrics (Prometheus), logs (Loki), and elasticsearch.

Named instances — add a __<NAME> suffix to the base URL var. Auth for a named instance uses the <PREFIX>__<NAME>_ prefix:

PROMETHEUS_URL=http://prom:9090                 # instance "default"
PROMETHEUS_URL__PROD=http://prom-prod:9090       # instance "PROD"
PROMETHEUS__PROD_AUTH_TOKEN=eyJhbGci...          # auth for "PROD"

Failover — any URL value may be a comma-separated list or JSON array. URLs are tried in order; the server fails over only on infrastructure errors (5xx / timeout / network) and never on a 4xx:

PROMETHEUS_URL=http://prom-a:9090,http://prom-b:9090

Rich form — MCP_BACKENDS (a JSON array) gives full control, including an explicit product (skips version auto-probe) and per-instance headers. It takes precedence over the env-var forms:

MCP_BACKENDS='[{"skill":"metrics","instance":"PROD",
  "urls":["http://mimir-a","http://mimir-b"],
  "authPrefix":"MIMIR_PROD","product":"Grafana Mimir",
  "extraHeaders":{"X-Scope-OrgID":"team-a"}}]'

Selecting a backend — tools on multi-backend skills accept an optional target argument naming the instance (e.g. "PROD"). Omit it to use the primary. target is validated against the configured instance names only, so a caller can never coerce the server into fetching an arbitrary URL (no SSRF).

Client Authentication (HTTP mode)

Clients connecting to the MCP server over HTTP must present an API key. Keys are loaded from (first match wins):

  1. MCP_AUTH_KEYS env var — JSON string (best for containers / K8s Secrets)

  2. MCP_AUTH_KEYS_FILE env var — path to a JSON file (K8s mounted Secret)

  3. ./auth-keys.json — local file in cwd

  4. ~/.otel-mcp/auth-keys.json — user home directory

If no keys are found, the server runs with open access (a warning is logged).

Key format:

{
  "keys": [
    {
      "id": "agent-1",
      "key": "sk-my-secret-key-here",
      "description": "Production RCA agent"
    },
    {
      "id": "ci-readonly",
      "key": "sk-ci-key",
      "description": "CI pipeline — restricted tools",
      "allowedTools": ["traces", "metrics"]
    }
  ]
}

Clients authenticate via either header:

  • Authorization: Bearer sk-my-secret-key-here

  • X-API-Key: sk-my-secret-key-here

The /health endpoint is always unauthenticated.

allowedTools is enforced at session creation: the MCP session only registers the listed skills, so out-of-scope tools do not exist for that session.

Just-in-Time (JIT) Privileged Identity

A static API key is standing privilege — whoever holds it holds it until someone rotates it. JIT identity treats the AI agent like a human employee: the static key becomes a role definition used only to sign in, and real access happens through short-lived, scoped, rotatable session tokens. Maps to the OWASP MCP Top 10 mitigations for MCP01 (token mismanagement), MCP02 (scope creep), MCP07 (authn/z), and MCP08 (audit & telemetry).

Enable with MCP_JIT_MODE:

Mode

Static keys

JIT tokens

off (default)

full access

— (/auth/* answers 404)

enabled

full access (migration)

accepted

required

may only mint tokens

required for all MCP calls

Token lifecycle (the "badge" flow):

# 1. Exchange the static key for a scoped, ephemeral token (default TTL 15 min)
curl -X POST http://localhost:3001/auth/token \
  -H "Authorization: Bearer sk-my-static-key" \
  -d '{"scopes": ["traces", "metrics"], "ttlSeconds": 600}'
# → 201 { "token": "mcpj_<id>.<secret>", "token_id": "<id>", "scopes": [...],
#         "expires_at": "...", "not_after": "...", "generation": 0 }

# 2. Call MCP with the ephemeral token instead of the key
curl -X POST http://localhost:3001/mcp \
  -H "Authorization: Bearer mcpj_..." ...

# 3. Rotate before expiry — same scopes, same session, next generation.
#    The old token stays valid for a 30 s grace window.
curl -X POST http://localhost:3001/auth/token/refresh \
  -H "Authorization: Bearer mcpj_..."

# 4. Hand the badge back (self), or kill-switch by id with a static key
curl -X POST http://localhost:3001/auth/token/revoke -H "Authorization: Bearer mcpj_..."
curl -X POST http://localhost:3001/auth/token/revoke \
  -H "Authorization: Bearer sk-admin-key" -d '{"token_id": "<id>"}'

Guarantees:

  • Least privilege — requested scopes must be a subset of the parent key's allowedTools (escalation → 403 scope_violation); omitting scopes inherits the parent's full grant. Tokens cannot mint further tokens, so every lineage is one auditable hop from a named key.

  • Ephemeral — per-token TTL is clamped to 60 s–1 h; rotation never extends a lineage past MCP_JIT_MAX_LIFETIME_SECONDS (default 8 h — end of shift), after which the agent must re-authenticate with its static key.

  • Session binding — an MCP session can only be reused by the credential lineage that created it (rotation preserves it; any other key or token gets 403). Tokens are held server-side as SHA-256 hashes only, and a restart invalidates everything outstanding.

  • Auditable — every issuance, rotation, revocation, and denial is logged with the public token id (never the secret) and exported on /metrics: mcp_jit_tokens_issued_total{parent_key}, mcp_jit_rotations_total, mcp_jit_revocations_total{source}, mcp_jit_denials_total{reason}, mcp_jit_active_tokens.

Configuration:

Env var

Default

Meaning

MCP_JIT_MODE

off

off | enabled | required

MCP_JIT_TTL_SECONDS

900

Token TTL; also the cap for requested ttlSeconds (60–3600)

MCP_JIT_MAX_LIFETIME_SECONDS

28800

Hard cap per token lineage (ttl–86400)

MCP_JIT_MAX_ACTIVE_TOKENS

1000

Active-token capacity guard

MCP_JIT_STORE

memory

State backend (see below)

Revocation has three forms: a token holder revokes itself by presenting the token; an operator revokes one generation with token_id; or an operator kills an entire rotation lineage with root_id (useful when a credential leaked and the current generation's id is unknown). A scope-restricted static key may only revoke tokens it minted — including after the lineage has rotated, where ownership is resolved from the lineage's live members rather than the (by then pruned) first-generation record. Unrestricted keys revoke anything.

Renewal re-checks the parent key, so access review cascades: remove a key and its live lineages can no longer refresh (they die at their current TTL); narrow a key's allowedTools and lineages holding wider scopes stop renewing.

JIT applies to the HTTP transport only — stdio is a local, single-user channel with no network credential to steal.

High availability (multiple replicas)

By default (MCP_JIT_STORE=memory) token state is in-process. That is the right failure mode for a single instance — a restart invalidates outstanding tokens and clients re-mint — but behind a load balancer it means a token minted on one replica is unknown on another, revocation only takes effect where it was processed, and an ID-JAG assertion could be redeemed once per replica.

For multi-replica deployments, point MCP_JIT_STORE at a module that exports createJitStore() returning { tokens, denylist } implementing the JitTokenStore and BoundedDenylist interfaces in src/jit-store.ts. Adapters are operator-supplied so no network client (Redis, a database driver) ever enters this package's dependencies. The interfaces are deliberately built from atomic compound primitives rather than get/set — insert performs the capacity check and the insert as one operation, rotate is a compare-and-set, and the denylist's addIfAbsent is a single check-and-record — so implement them with your backend's transactional facility (MULTI, a Lua script, a conditional write). A get/set implementation would reintroduce cross-replica races on the capacity cap, once-only rotation, and assertion single-use.

MCP sessions are a separate matter: they hold live transport objects and cannot be shared. Use sticky routing on Mcp-Session-Id, or the stateless transport mode, so any replica can serve any request. The principal binding is recomputed per request and survives either choice.

Enterprise-Managed Authorization (MCP extension)

Implements the server side of the MCP Enterprise-Managed Authorization extension (io.modelcontextprotocol/enterprise-managed-authorization): your identity provider (Okta, Entra ID, corporate SSO) decides who may use this MCP server, instead of locally provisioned API keys. The MCP client logs the employee in via SSO, obtains an ID-JAG (Identity Assertion JWT Authorization Grant) from the IdP, and exchanges it at this server's token endpoint for a scoped JIT session token — enterprise identities ride the same least-privilege, rotation, and audit rails as key-minted tokens. Onboarding, offboarding, and access policy live in the IdP admin console.

MCP_ENTERPRISE_AUTH_ISSUER=https://idp.example.com      # trusted IdP (required)
MCP_ENTERPRISE_AUTH_AUDIENCE=https://mcp.example.com    # this server's issuer id (required)
# optional:
MCP_ENTERPRISE_AUTH_RESOURCE=https://mcp.example.com    # resource id (default: audience)
MCP_ENTERPRISE_AUTH_JWKS_URL=https://idp.example.com/jwks  # default: OIDC discovery
MCP_ENTERPRISE_AUTH_DEFAULT_SCOPES=traces,metrics       # when the ID-JAG has no scope claim
MCP_ENTERPRISE_AUTH_MAX_REDEEMED_JTIS=50000             # replay-cache cap (fail-closed)

Configuring it auto-enables the JIT token infrastructure (and counts as client auth being "on" — an enterprise-only deployment is never open), and:

  • Discovery — serves RFC 8414 metadata at /.well-known/oauth-authorization-server advertising urn:ietf:params:oauth:grant-profile:id-jag in authorization_grant_profiles_supported, plus RFC 9728 /.well-known/oauth-protected-resource; MCP 401s carry a WWW-Authenticate: Bearer resource_metadata="…" challenge.

  • Token exchange — POST /auth/token with grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer and the ID-JAG as assertion (form-encoded or JSON) answers with an RFC 6749 token response whose access_token is a JIT session token scoped to the assertion's scope claim (intersected with enabled skills).

  • Validation per the ID-JAG profile — typ: oauth-id-jag+jwt; RS256/384/512, ES256/384, or EdDSA (Ed25519) signature against the IdP JWKS (cached, kid-aware, stale-cache fallback); iss/aud/resource binding; exp/iat/nbf with 60 s skew; single-use jti (replay → invalid_grant; the replay cache is bounded by MCP_ENTERPRISE_AUTH_MAX_REDEEMED_JTIS, shared across replicas when MCP_JIT_STORE is configured, and surfaced as the mcp_jit_idjag_replay_cache_size gauge); client_id claim/request binding. Errors are RFC 6749 §5.2 token errors, counted in mcp_jit_denials_total{reason="idjag_*"}. If token issuance fails after the assertion verified, the jti is un-redeemed so the client can retry the same assertion rather than round-tripping the IdP.

  • Account linking — the assertion sub is the audit identity (logged on issuance); tokens mint under parent_key="enterprise-idp" in metrics.

# Client-side exchange (the MCP client does this automatically)
curl -X POST https://mcp.example.com/auth/token \
  -d grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer \
  -d assertion="$ID_JAG" -d client_id=my-mcp-client
# → { "access_token": "mcpj_…", "token_type": "Bearer", "expires_in": 900, "scope": "metrics traces" }

MCP 2026-07-28 support

The 2026-07-28 revision made MCP stateless: the initialize handshake and Mcp-Session-Id are gone, so any request can land on any server instance. This server implements that revision and keeps serving ≤2025-11-25 clients on the same endpoint — the revision is resolved per request, from the MCP-Protocol-Version header or (when absent) from the presence of the 2026 routing headers or _meta client info.

Change

Status

Stateless core — no handshake, no session id (SEP-2575, SEP-2567)

A 2026 client's first request can be a bare tools/call; no session id is issued and none is required. Only POST applies — there is no standalone SSE stream and no session to delete, so GET/DELETE get 405

Mcp-Method / Mcp-Name routing headers (SEP-2243)

Validated against the request body; disagreement is rejected with JSON-RPC -32600

server/discover (SEP-2575)

Returns server info, capabilities, supported revisions, and the skill inventory

ttlMs / cacheScope cache hints (SEP-2549)

On tools/list, resources/list, and resources/read. Tool calls are never marked cacheable — they are live telemetry queries

W3C Trace Context in _meta (SEP-414)

traceparent/tracestate/baggage propagate to every backend query, so one trace spans host → MCP server → Prometheus/Jaeger/Loki

Extensions map (SEP-2133)

Enterprise-managed authorization is declared as io.modelcontextprotocol/enterprise-managed-authorization when configured

Resource-not-found is -32602 (SEP-2164)

Already emitted; covered by a regression test

Roots / Sampling / Logging deprecations

Never used — diagnostics go to stderr and /metrics, which is the replacement the spec points to

Trace propagation is the piece worth calling out: a tool call and the backend queries it triggers used to be unrelated spans. With a traceparent in the call's _meta, they now join up in any OpenTelemetry-compatible backend. If a proxy stamps HTTP trace headers instead, those are used as a fallback; _meta wins when both are present, and malformed values are dropped rather than forwarded.

GET /health reports the supported revisions, and /metrics carries mcp_spec_requests_total{version,mode}, mcp_trace_context_propagated_total{source}, and mcp_routing_header_rejections_total{header}.

On the SDK: @modelcontextprotocol/sdk v1 tops out at protocol 2025-11-25 and rejects a newer version header outright, so this support is implemented in this server's HTTP layer above the SDK: the body is buffered once (which is what makes the header/body cross-check possible), the version header is translated to one the SDK accepts, and 2026 requests are served from a fresh session-less transport per request. When an SDK with native support ships, that shim collapses; the behaviour clients see does not change.

Kubernetes Deployment

apiVersion: v1
kind: Secret
metadata:
  name: otel-mcp-auth
stringData:
  # Client keys
  auth-keys.json: |
    {"keys":[{"id":"rca-agent","key":"sk-prod-xxx"}]}
  # Backend tokens
  PROMETHEUS_AUTH_TOKEN: "my-prom-token"
  LOKI_AUTH_TOKEN: "my-loki-token"
  LOKI_TENANT_ID: "platform"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-mcp-server
spec:
  # Safe to scale past 1 once every client speaks MCP 2026-07-28 (stateless —
  # any replica serves any request). While pre-2026 clients remain, either keep
  # one replica or enable sticky routing on Mcp-Session-Id at the ingress; and
  # if JIT identity is enabled, set MCP_JIT_STORE to a shared adapter so tokens
  # and revocations are visible across replicas. See "High availability".
  replicas: 1
  template:
    spec:
      containers:
        - name: otel-mcp-server
          image: moebiusx/otel-mcp-server:latest
          ports:
            - containerPort: 3001
          env:
            - name: JAEGER_URL
              value: "http://jaeger-query.observability:16686"
            - name: PROMETHEUS_URL
              value: "http://prometheus.observability:9090"
            - name: LOKI_URL
              value: "http://loki.observability:3100"
            # Optional: uncomment to enable Elasticsearch / Alertmanager / Grafana skills
            # - name: ELASTICSEARCH_URL
            #   value: "http://elasticsearch.observability:9200"
            # - name: ALERTMANAGER_URL
            #   value: "http://alertmanager.observability:9093"
            # - name: GRAFANA_URL
            #   value: "http://grafana.observability:3000"
            # - name: GRAFANA_AUTH_TOKEN
            #   valueFrom:
            #     secretKeyRef:
            #       name: otel-mcp-auth
            #       key: GRAFANA_AUTH_TOKEN
            - name: PROMETHEUS_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: otel-mcp-auth
                  key: PROMETHEUS_AUTH_TOKEN
            - name: LOKI_AUTH_TOKEN
              valueFrom:
                secretKeyRef:
                  name: otel-mcp-auth
                  key: LOKI_AUTH_TOKEN
            - name: LOKI_TENANT_ID
              valueFrom:
                secretKeyRef:
                  name: otel-mcp-auth
                  key: LOKI_TENANT_ID
            - name: MCP_AUTH_KEYS_FILE
              value: "/etc/otel-mcp/auth-keys.json"
          volumeMounts:
            - name: auth-keys
              mountPath: /etc/otel-mcp
              readOnly: true
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 3001
            initialDelaySeconds: 5
          readinessProbe:
            httpGet:
              path: /health
              port: 3001
      volumes:
        - name: auth-keys
          secret:
            secretName: otel-mcp-auth
            items:
              - key: auth-keys.json
                path: auth-keys.json

Skills

Each telemetry backend is a skill — an independent plugin. The core OTel skills (traces, metrics, logs) and the app skills (system, zk-proofs, public-exchange) are always active, defaulting to localhost backends so the server works out of the box. Every other skill is opt-in: it activates only when its backend URL is set (e.g. ELASTICSEARCH_URL, CILIUM_URL, CLICKHOUSE_URL), and is silently skipped otherwise. Use --tools to restrict which skills load regardless of configuration.

Traces — traces — 5 tools

Provider-agnostic. Select backend with TRACES_PROVIDER (jaeger [default], tempo, zipkin, skywalking). The verb surface is stable across providers; capabilities the chosen backend doesn't support (e.g. traces_dependencies on Tempo) return a clear error.

Tool

Description

traces_search

Search traces by service, operation, tags, or duration

trace_get

Full trace detail — all spans with timing, tags, and parent-child

traces_services

List all reporting services

traces_operations

List operations for a service

traces_dependencies

Service dependency graph

Metrics (Prometheus) — metrics — 6 tools

Multi-backend: every tool accepts an optional target argument to select a named instance (see Multi-backend instances & failover).

Tool

Description

metrics_query

Instant PromQL query

metrics_query_range

Range PromQL query (time series)

metrics_targets

Scrape target health

metrics_alerts

Alerting rules and state

metrics_metadata

Metric type, help, unit lookup

metrics_label_values

Label value enumeration

Logs (Loki) — logs — 4 tools

Multi-backend: every tool accepts an optional target argument to select a named instance (see Multi-backend instances & failover).

Tool

Description

logs_query

LogQL query for log lines

logs_labels

Available label names

logs_label_values

Values for a label

logs_tail_context

Logs correlated with a trace ID

Enabled when ELASTICSEARCH_URL is set. Multi-backend: every tool accepts an optional target argument to select a named instance (see Multi-backend instances & failover).

Tool

Description

es_search

Full-text search across indices with Lucene query syntax

es_cluster_health

Cluster health (green/yellow/red), node and shard counts

es_indices

List indices with doc counts, storage size, and health

es_index_mapping

Field mappings, types, and analyzers for an index

es_cat_nodes

Node resource usage (CPU, heap, disk, load)

Alertmanager — alertmanager — 4 tools

Enabled when ALERTMANAGER_URL is set.

Tool

Description

alertmanager_alerts

Active alerts with labels, annotations, and routing status

alertmanager_silences

List active/pending/expired silences with matchers

alertmanager_groups

Alert groups by routing rules and receivers

alertmanager_status

Cluster status, version, peer count, and live config

vmalert — vmalert — 4 tools

Enabled when VMALERT_URL is set (e.g. http://localhost:8880). vmalert is the rule-evaluation component in a VictoriaMetrics stack — VM single-node stores series but does not evaluate rules, so use this skill to query rules and active alerts from vmalert directly.

Tool

Description

vmalert_rules

Alerting and recording rules with state, query, and evaluation health. Filterable by type (all/alerting/recording) and state (all/firing/pending/inactive)

vmalert_alerts

Active alerts as vmalert sees them pre-Alertmanager, with labels, value, and deep-link source

vmalert_groups

Rule groups with interval, concurrency, and rule counts by type

vmalert_rule_health

Rules whose evaluation health is not ok — surfaces evaluation errors immediately

Grafana — grafana — 10 tools

Enabled when GRAFANA_URL is set. The 10 tools below are read-only and intended for verification/interrogation workflows. Three additional write tools are available when MCP_ENABLE_WRITES is set — see Write tools.

Tool

Description

grafana_health

Grafana health, version, commit, and database status

grafana_datasources

List data sources with safe metadata

grafana_datasource_health

Check one data source by UID

grafana_datasource_query

Run read-only queries through Grafana's unified data source query API

grafana_dashboards_search

Search dashboards and folders by text, tag, folder, type, or starred status

grafana_dashboard_get

Get dashboard structure, panels, variables, data source references, and panel queries

grafana_folders

List folders with UID, title, URL, and metadata

grafana_alert_rules

List Grafana-managed alert rules and query references

grafana_alerts

List active Grafana Alertmanager alert instances

grafana_contact_points

List alert contact points or receivers with safe integration status metadata

Write tools (opt-in)

Disabled by default. The server is read-only out of the box. Set MCP_ENABLE_WRITES=true (also accepts 1/yes/on) to advertise and enable the mutating tools below. The Grafana token must also carry the matching write scopes.

Tool

Description

Token scope

grafana_create_dashboard

Create / upsert / update a dashboard (POST /api/dashboards/db)

dashboards:write

grafana_delete_dashboard

Delete a dashboard by UID (DELETE /api/dashboards/uid/{uid})

dashboards:delete

grafana_create_folder

Create or upsert a folder

folders:write

grafana_create_alert_rule

Create / upsert / update a Grafana-managed alerting or recording rule (/api/v1/provisioning/alert-rules)

alert.provisioning:write

grafana_delete_alert_rule

Delete a Grafana-managed rule by UID (DELETE /api/v1/provisioning/alert-rules/{uid})

alert.provisioning:write

Write modes — each write tool takes an explicit mode so the caller controls overwrite behavior; the default is the safe one:

  • create (default) — strict insert: fails with a clear conflict error (including the existing object's UID and version) if the target UID already exists. Use this for promotion workflows (e.g. staging → prod) where silently overwriting is dangerous.

  • upsert — idempotent create-or-update by UID, for reconcile / infra-as-code loops.

  • update (dashboards and alert rules) — strict update: fails if the target UID does not already exist.

Alert rules are Grafana-managed (the JSON provisioning API, no YAML dependency). A rule whose body includes a record object is treated as a recording rule; otherwise it is an alerting rule. Provisioning writes are sent with X-Disable-Provenance: true so the rules stay editable in the Grafana UI. (Mimir/Cortex ruler rules are YAML-based and remain a future follow-up.)

All write tools accept dry_run: true to validate and report the planned action without writing. Conflict detection for strict create/update uses a GET pre-check, and grafana_create_dashboard also sends Grafana's native overwrite=false as a second safety net. Returns the resulting UID and version on success. Existing read-only behavior is unchanged when writes are disabled.

Cilium (eBPF networking) — cilium — 6 tools

Enabled when CILIUM_URL is set. Targets the cilium-agent REST API. This is the agent control-plane surface; L3/L7 flow observability (Hubble) is gRPC and not yet wired.

Tool

Description

cilium_health

Agent datapath/controller status, kube-apiserver and kvstore connectivity

cilium_endpoints

Managed endpoints (pods) with security identity, state, and addressing

cilium_identities

Security identities — the numeric identity each label set maps to

cilium_policy

Network policy currently enforced, with revision

cilium_services

eBPF load-balancing services and their backends

cilium_nodes

Nodes known to the agent (incl. cluster-mesh peers)

Grafana Beyla (eBPF auto-instrumentation) — beyla — 4 tools

Enabled when BEYLA_PROMETHEUS_URL is set. Beyla has no query API of its own — it auto-instruments processes with eBPF (zero code changes) and exports OTel/Prometheus metrics, so point this at the Prometheus-compatible store that scrapes Beyla. The tools issue Beyla-aware PromQL so you don't need to know its metric/label schema.

Tool

Description

beyla_services

Services Beyla is auto-instrumenting, with each one's request rate

beyla_red_metrics

RED for a service — request rate, error %, and p50/p95/p99 latency

beyla_top_routes

Busiest HTTP routes for a service, with per-route rate and p95

beyla_network_flows

Top service-to-service network flows by throughput (eBPF network metrics)

Kubernetes (CRD reader) — kubernetes — 5 tools

Auto-enabled in-cluster (ServiceAccount mount), or set KUBERNETES_URL + token out-of-cluster. Read-only (GET only). The generic k8s_list/k8s_get work for any built-in resource or CRD, so the whole control-plane tier (Argo Rollouts, Flagger, Kyverno, Gatekeeper, KEDA, Chaos Mesh, Cilium policies, Inspektor Gadget, …) is queryable without a bespoke skill per product.

Tool

Description

k8s_health

kube-apiserver connectivity — server version and readiness

k8s_api_resources

Discover installed API groups / CRD kinds (find what's installed)

k8s_list

List objects of any resource or CRD, with curated status

k8s_get

Get a single object by name, with full status and optional spec

k8s_events

Recent cluster events, filtered by namespace and type

ClickHouse Logs — clickhouse — 5 tools

Enabled when CLICKHOUSE_URL is set. Uses ClickHouse's HTTP GET query path, which the engine forces to be read-only — writes are rejected by ClickHouse itself.

Tool

Description

clickhouse_query

Run a read-only SQL query (SELECT/SHOW/DESCRIBE) with column types

clickhouse_databases

List databases

clickhouse_tables

List tables with engine and approximate row/byte counts

clickhouse_table_schema

Describe a table — columns, types, codecs

clickhouse_logs_search

Convenience log search — time window, message ILIKE, level, newest first

Pyroscope (continuous profiling) — pyroscope — 4 tools

Enabled when PYROSCOPE_URL is set. Works against OSS Pyroscope and Grafana Pyroscope.

Tool

Description

pyroscope_profile_types

List available profile/application names

pyroscope_labels

Label names available for a profile type

pyroscope_label_values

Values for a given label

pyroscope_render

Render a profile and return the heaviest functions by self time

Open Policy Agent — opa — 4 tools

Enabled when OPA_URL is set. Read-only (GET against the Data/Query APIs).

Tool

Description

opa_health

OPA health, including bundle activation

opa_policies

Loaded policy modules with package paths

opa_data

Fetch/evaluate a document at a data path, with optional input

opa_query

Ad-hoc Rego query — e.g. enumerate violations across packages

Envoy — envoy — 4 tools

Enabled when ENVOY_ADMIN_URL is set. Works for standalone Envoy and mesh sidecar proxies.

Tool

Description

envoy_server_info

Version, serving state, and uptime

envoy_clusters

Upstream clusters and per-endpoint health

envoy_listeners

Configured listeners and bind addresses

envoy_stats

Counters/gauges, optionally filtered by name regex

Consul — consul — 5 tools

Enabled when CONSUL_URL is set. Set CONSUL_AUTH_TOKEN for an ACL token.

Tool

Description

consul_health

Agent datacenter, node, version, role, and current leader

consul_services

Registered services with tags

consul_service_instances

Instances of a service with address, port, and health

consul_checks

Health checks in a given state (defaults to critical)

consul_members

Cluster members and gossip status

Kong Gateway — kong — 4 tools

Enabled when KONG_ADMIN_URL is set.

Tool

Description

kong_status

Node version, database reachability, connection stats

kong_services

Configured services (upstream targets)

kong_routes

Routes and the services they map to

kong_plugins

Enabled plugins and their scope

Traefik — traefik — 4 tools

Enabled when TRAEFIK_URL is set.

Tool

Description

traefik_overview

Version and router/service/middleware counts and features

traefik_routers

HTTP routers — rules, target service, status, entry points

traefik_services

HTTP services — type, status, load-balancer server health

traefik_entrypoints

Configured entry points and bind addresses

InfluxDB — influx — 3 tools

Enabled when INFLUX_URL is set. Uses the InfluxQL /query endpoint (1.x and 2.x compatible).

Tool

Description

influx_health

Health, status, and version

influx_databases

List databases / DBRP-mapped buckets

influx_query

Run a read-only InfluxQL query and return series

OpenTSDB — opentsdb — 3 tools

Enabled when OPENTSDB_URL is set.

Tool

Description

opentsdb_version

Version and build info

opentsdb_suggest

Autocomplete metric names, tag keys, or tag values

opentsdb_query

Query a metric over a range with aggregator, downsampling, and tag filters

Graylog — graylog — 3 tools

Enabled when GRAYLOG_URL is set.

Tool

Description

graylog_system

Node version, lifecycle state, hostname, start time

graylog_streams

Streams (message routing rules)

graylog_search

Search messages over a relative time window (Graylog query syntax)

Grafana Tempo, Apache SkyWalking — see traces

Tempo and SkyWalking are now exposed through the provider-agnostic traces skill — set TRACES_PROVIDER=tempo or TRACES_PROVIDER=skywalking and point the matching URL var at your backend.

Pinpoint — pinpoint — 3 tools

Enabled when PINPOINT_URL is set. The API varies by Pinpoint version, so a read-only GET passthrough is provided for version-specific endpoints.

Tool

Description

pinpoint_applications

Monitored applications and service types

pinpoint_server_time

Current server time (for building time ranges)

pinpoint_get

Read-only GET against any Pinpoint API path

Collection Pipelines — pipeline — 4 tools

Enabled when any of FLUENTBIT_URL / BEATS_URL / VECTOR_URL / ALLOY_URL is set. Each tool errors clearly if its agent isn't configured.

Tool

Description

pipeline_fluentbit

Fluent Bit per-input/output records, bytes, retries, drops

pipeline_beats

Beats output event throughput and write errors

pipeline_vector

Vector health and configured components

pipeline_alloy

Grafana Alloy components and their health

ZK Proofs — zk-proofs — 4 tools

Tool

Description

zk_proof_get

Retrieve a ZK-SNARK proof

zk_proof_verify

Verify a proof server-side

zk_solvency

Latest solvency proof

zk_stats

Aggregate proof statistics

System — system — 5 tools

Tool

Description

anomalies_active

Active anomalies

anomalies_baselines

Detection baselines

system_health

Full health check

system_topology

Service dependency topology

backend_capabilities

Supported product versions and per-feature availability; optionally classifies a concrete version into its support tier and reports the active gating mode

AgentRelay — agentrelay — 1 tool (+1 write tool via MCP_ENABLE_WRITES)

Enabled when AGENTRELAY_URL is set (e.g. https://api.agentrelay.tech). Set AGENTRELAY_AUTH_TOKEN for a bearer token, or AGENTRELAY_AUTH_HEADER to send a raw header value (e.g. an X-API-Key scheme).

Coordinate with other agents through the AgentRelay hosted REST API ("headless Slack for agents"). Read-only by default; the send tool is opt-in behind MCP_ENABLE_WRITES.

Tool

Description

agentrelay_agents

List the agents currently connected to your AgentRelay organization

agentrelay_send

(write) Send a message or task to another agent via POST /v1/relay/send. Requires MCP_ENABLE_WRITES. Supports dry_run

Public Exchange — public-exchange — 5 tools

Always available (only needs APP_API_URL, which defaults to http://localhost:5000). Read-only tools mirroring KrystalineX's /api/public/* transparency endpoints — designed for an unauthenticated public MCP deployment, since every endpoint already serves data that is public on the transparency website.

Tool

Description

exchange_status

Operational state, uptime, observability posture

total_volume

Aggregate 24h / weekly / all-time trading volume

recent_trades

Anonymized recent trades feed (limit ≤ 100)

transparency_metrics

Full transparency metrics bundle

verify_trace

Public-safe distributed trace for a trade ID

Selective Skills

Only load the skills you need:

# Core OTEL only (no ZK / system health)
node dist/index.js --tools traces,metrics,logs

# Traces + metrics + alertmanager
node dist/index.js --http 3001 --tools traces,metrics,alertmanager

Self-Metrics

In HTTP mode, GET /metrics exposes Prometheus-format metrics about the MCP server itself:

Metric

Type

Description

mcp_tool_calls_total{tool,status}

Counter

Tool invocation count

mcp_tool_duration_seconds{tool}

Histogram

Tool call latency

mcp_tool_errors_total{tool}

Counter

Tool call errors

mcp_backend_requests_total{backend,status}

Counter

Outbound backend HTTP requests

mcp_backend_duration_seconds{backend}

Histogram

Backend request latency

mcp_auth_attempts_total{result}

Counter

Client auth attempts (accepted/rejected)

mcp_active_sessions

Gauge

Currently connected MCP sessions (pre-2026 clients)

mcp_uptime_seconds

Gauge

Server uptime

mcp_server_info{version}

Gauge

Server version metadata (constant 1)

mcp_build_info{service,version,sha,ref,built_at}

Gauge

Build provenance of the running image (constant 1)

Protocol and request shape — use these to see which MCP revision clients speak, and whether callers are propagating trace context:

Metric

Type

Description

mcp_spec_requests_total{version,mode}

Counter

Requests by negotiated spec revision and lifecycle (stateless / session). Watch the mix to know when every client has moved to 2026-07-28

mcp_trace_context_propagated_total{source}

Counter

Inbound requests by trace-context source (meta / http / none)

mcp_routing_header_rejections_total{header}

Counter

Requests rejected because Mcp-Method/Mcp-Name disagreed with the body — a nonzero rate means a broken gateway or an attempted smuggle

JIT privileged identity (see above) — only populated when MCP_JIT_MODE is set or enterprise auth is configured:

Metric

Type

Description

mcp_jit_tokens_issued_total{parent_key}

Counter

Session tokens minted, by parent key

mcp_jit_rotations_total

Counter

Token rotations (refresh grants)

mcp_jit_revocations_total{source}

Counter

Revocations, by source (self / admin)

mcp_jit_denials_total{reason}

Counter

Denials by reason (expired, revoked, scope_violation, parent_key_revoked, idjag_*, …)

mcp_jit_active_tokens

Gauge

Active (unexpired, unrevoked) tokens

mcp_jit_idjag_replay_cache_size

Gauge

Remembered redeemed ID-JAG jtis

Scrape with Prometheus:

scrape_configs:
  - job_name: 'otel-mcp-server'
    static_configs:
      - targets: ['otel-mcp-server:3001']

Client Integration

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "otel": {
      "command": "npx",
      "args": ["-y", "@moebiusx/otel-mcp-server"],
      "env": {
        "JAEGER_URL": "http://localhost:16686",
        "PROMETHEUS_URL": "http://localhost:9090",
        "LOKI_URL": "http://localhost:3100"
      }
    }
  }
}

VS Code / GitHub Copilot

Add to .vscode/mcp.json:

{
  "servers": {
    "otel": {
      "command": "npx",
      "args": ["-y", "@moebiusx/otel-mcp-server"],
      "env": {
        "JAEGER_URL": "http://localhost:16686",
        "PROMETHEUS_URL": "http://localhost:9090",
        "LOKI_URL": "http://localhost:3100"
      }
    }
  }
}

HTTP Client (any agent)

# Health check — skills, backend versions, supported MCP revisions
curl -s http://localhost:3001/health

# List tools. MCP 2026-07-28 is stateless, so this is a single POST:
# no initialize handshake, no session id.
curl -s -X POST http://localhost:3001/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# Add the credential when MCP_AUTH_KEYS is set
#   -H "Authorization: Bearer sk-my-key"

Older clients that send initialize and carry an Mcp-Session-Id are served on the same endpoint unchanged — see MCP 2026-07-28 support. Omitting MCP-Protocol-Version puts the request on the pre-2026 path, where a bare tools/list is rejected with Server not initialized until the handshake completes.

Architecture

Each telemetry backend is a Skill — a self-contained plugin that declares its tools, self-configures from env vars, and registers MCP tools on the server.

src/
├── index.ts              # CLI entry point (stdio / HTTP transport)
├── server.ts             # MCP server factory (iterates skills)
├── skill.ts              # Skill interface + SkillHelpers factory
├── skills.ts             # Skill registry wrapper (+ version metadata)
├── skills.generated.ts   # GENERATED by `npm run gen:skills` — do not edit
├── config.ts             # env() helper
├── auth.ts               # Backend + client authentication
├── oauth.ts              # Backend OAuth 2.0 client-credentials
├── jit.ts                # JIT privileged identity — scoped ephemeral tokens
├── jit-store.ts          # Pluggable token/denylist store (MCP_JIT_STORE) — HA
├── enterprise-auth.ts    # Enterprise-managed authorization (MCP ext-auth ID-JAG)
├── mcp-spec.ts           # MCP spec revisions + per-revision feature flags
├── request-context.ts    # Per-request context (W3C trace propagation)
├── helpers.ts            # fetchJSON, createFetcher, utilities
├── metrics.ts            # Self-metrics (Prometheus format)
├── versions.ts           # Backend version-support model + feature gates
├── version-registry.ts   # Live backend version detection (TTL-cached)
├── protocols.ts          # Query-protocol catalog (PromQL, LogQL, …)
├── gating.ts             # MCP_VERSION_GATING enforcement
├── compat.ts             # Tool alias / argument-mapping facade
├── tools/
│   ├── traces.ts         # Traces layer — dispatches to a provider per TRACES_PROVIDER (5 tools)
│   ├── metrics.ts        # Prometheus metrics skill (6 tools)
│   ├── logs.ts           # Loki logs skill (4 tools)
│   ├── elasticsearch.ts  # ES/OpenSearch skill (5 tools)
│   ├── alertmanager.ts   # Alertmanager skill (4 tools)
│   ├── grafana.ts        # Grafana read-only skill (10 tools)
│   ├── cilium.ts         # Cilium eBPF networking skill (6 tools)
│   ├── beyla.ts          # Grafana Beyla eBPF auto-instrumentation skill (4 tools)
│   ├── kubernetes.ts     # Kubernetes CRD reader skill (5 tools)
│   ├── clickhouse.ts     # ClickHouse logs skill (5 tools)
│   ├── pyroscope.ts      # Pyroscope profiling skill (4 tools)
│   ├── opa.ts            # Open Policy Agent skill (4 tools)
│   ├── envoy.ts          # Envoy proxy admin skill (4 tools)
│   ├── consul.ts         # Consul skill (5 tools)
│   ├── kong.ts           # Kong Gateway skill (4 tools)
│   ├── traefik.ts        # Traefik skill (4 tools)
│   ├── influxdb.ts       # InfluxDB metrics skill (3 tools)
│   ├── opentsdb.ts       # OpenTSDB metrics skill (3 tools)
│   ├── graylog.ts        # Graylog logs skill (3 tools)
│   ├── pinpoint.ts       # Pinpoint skill (3 tools)
│   ├── pipeline.ts       # Collection pipelines skill (4 tools)
│   ├── zk-proofs.ts      # ZK proof skill (4 tools)
│   └── system.ts         # System health skill (4 tools)
├── providers/
│   └── traces/           # Trace provider implementations (Layer→Provider pattern)
│       ├── types.ts      # TracesProvider interface + factory type
│       ├── jaeger.ts     # Jaeger Query API provider (default)
│       ├── tempo.ts      # Grafana Tempo TraceQL provider
│       ├── zipkin.ts     # Zipkin v2 provider
│       └── skywalking.ts # SkyWalking OAP GraphQL provider
├── resources/
│   └── overview.ts       # MCP resource: auto-generated overview
└── transports/
    ├── session-store.ts  # HTTP session lifecycle + principal binding (pre-2026)
    ├── mcp-2026.ts       # MCP 2026-07-28 request handling (stateless, headers)
    └── jit-endpoints.ts  # /auth/token mint · refresh · revoke handlers

Adding a new skill

// 1. Create src/tools/tempo.ts
import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import type { Skill, SkillHelpers } from '../skill.js';

function registerTools(server: McpServer, helpers: SkillHelpers): void {
  const tempoUrl = helpers.env('TEMPO_URL');
  const fetchJSON = helpers.createFetcher('TEMPO', 'tempo');

  server.tool('tempo_search', 'Search traces in Tempo', { ... }, async (params) => {
    // ...
  });
}

export const skill: Skill = {
  id: 'tempo',
  name: 'Grafana Tempo',
  description: 'Query traces via the Grafana Tempo API',
  tools: 1,
  backends: ['Tempo'],
  isAvailable: () => !!process.env.TEMPO_URL,
  register: registerTools,
};

// 2. Add to src/skills.ts
import { skill as tempo } from './tools/tempo.js';
export const allSkills: Skill[] = [...existingSkills, tempo];

Auth Flow

Client → [API Key] → MCP Server → [Backend Credentials] → Jaeger/Prometheus/Loki
                          │
                          ├── Authorization: Bearer <JAEGER_AUTH_TOKEN>  → Jaeger
                          ├── Authorization: Basic <PROMETHEUS_AUTH_BASIC> → Prometheus
                          └── Authorization: Bearer <LOKI_AUTH_TOKEN>    → Loki
                               X-Scope-OrgID: <LOKI_TENANT_ID>

With JIT identity enabled, the static key only signs in; tools are reached with a scoped, expiring token (see Just-in-Time (JIT) Privileged Identity):

Client → [API Key] → POST /auth/token ──► mcpj_… (scoped, TTL, rotatable)
Client → [mcpj_…]  → MCP Server (session bound to the token lineage)
                          └── refresh before expiry / revoke when done

Development

# Dev mode (tsx, no build step)
npm run dev             # stdio
npm run dev:http        # HTTP on port 3001

# Type check
npm run lint

# Build
npm run build

# Tests
npm test

# Tests with coverage (thresholds enforced on the auth surface)
npm run test:coverage

# Run a single test file
npx vitest run tests/auth.test.ts

# Docker live tests against local backend fixtures
npm run test:live
node scripts/live-test.mjs --skill metrics

For the Docker-backed one-skill-at-a-time workflow, fixture coverage, report viewer, and troubleshooting, see docs/live-testing.md.

Appendix: Live Cluster Analysis

The following analysis was generated entirely by an AI agent (GitHub Copilot CLI) using this MCP server to query a production KrystalineX cluster — 27 tool calls across 6 skills, zero manual commands. This is what "Proof of Observability" looks like in practice.

Cluster: KrystalineX crypto exchange · 3-node K8s (1 control-plane, 2 workers) · Helm-managed
MCP Server: v1.2.0 · 6/7 skills active (Elasticsearch disabled) · session-based HTTP transport
Date: 2026-03-24T19:30 UTC


Infrastructure

Node

Role

CPU

Memory

Disk

Status

kube (192.168.1.32)

control-plane

13.4%

29.7%

62.3%

✅ Ready

worker1 (192.168.1.34)

worker

10.0%

28.6%

51.2%

✅ Ready

worker2 (192.168.1.35)

worker

—

—

—

🔴 NotReady (hardware)

Prometheus Targets — 12/12 UP

All scrape targets healthy with zero errors:

krystalinex-server · payment-processor · Kong · Grafana · Jaeger · OTEL Collector · Prometheus · RabbitMQ · Redis · kube-state-metrics · node-exporter ×2

Application Services

Service

Avg Latency

Traced Spans

Anomalies

Status

kx-exchange

491 ms

19

0

✅ Healthy

kx-wallet

156 ms

6

0

✅ Healthy

kx-matcher

29 ms

3

0

✅ Healthy

kx-gateway

—

(traced)

0

✅ Healthy

Performance Snapshot

Metric

Value

Threshold

Verdict

P50 latency

4.1 ms

—

🟢 Excellent

P99 latency

424 ms

2 s

🟢 Well within budget

Error rate (5xx)

0%

5%

🟢 Clean

Request throughput

0.21 req/s

—

Idle / low traffic

RabbitMQ backlog

0 messages

—

🟢 No queuing

Pod restarts

0

—

🟢 Stable

SLO Error Budgets

SLO

Budget Remaining

Status

Availability (99.9% target)

100%

🟢 Full

Latency (P99 < 2s target)

14.2%

🟡 Predictive alert firing

The latency SLO budget is being consumed faster than expected. A predict_linear rule forecasts exhaustion within 24 hours. Worth investigating tail latency in kx-exchange.

Active Alerts — 3

Alert

Severity

Detail

PodNotReady

⚠️ warning

server-df98765f9-pcxxv — stale pod from rollout, auto-resolving

PodNotReady

⚠️ warning

payment-processor-f4f8b8d78-wxszp — stale pod, auto-resolving

LatencyBudgetExhaustion

⚠️ warning

Predictive: latency error budget depleting within 24h

All critical rules — HighErrorRate, ServiceDown, ContainerCrashLooping, OOMKilled — are inactive.

Service Dependency Graph

kx-gateway ──(99 calls)──► kx-exchange ──(5 calls)──► kx-matcher
                                │
                                └──(94 calls)──► jaeger (OTEL export)

kx-wallet  ──(23 calls)──► kx-exchange
kx-wallet  ──(14 calls)──► kx-gateway

Logs & Traces

Signal

Window

Count

Finding

Error logs

1 h

0

Clean

Warning logs

1 h

0

Clean

OOM logs

6 h

0

No memory pressure

Error traces

1 h

3

Transient tcp.connect / dns.lookup — network hiccups

Slow traces (>2s)

1 h

0

No significant slow requests

Tools Used

This analysis invoked 27 MCP tool calls:

Skill

Tools Called

Traces

traces_services, traces_search ×2, traces_dependencies, system_health

Metrics

metrics_query ×12 (up, latency, errors, CPU, memory, disk, SLO budgets, RabbitMQ), metrics_targets, metrics_alerts

Logs

logs_query ×3 (errors, warnings, OOM)

Alertmanager

alertmanager_alerts, alertmanager_groups, alertmanager_status

ZK Proofs

zk_stats

System

anomalies_active, anomalies_baselines

Verdict

The cluster is healthy and stable with generous headroom on both active nodes. The main items to watch are the latency SLO budget trend and kube node disk usage at 62%. The offline worker2 node is a known hardware issue. No action required on the PodNotReady alerts — they are ephemeral artifacts of recent deployments.


Monorepo Integration (git subtree)

This directory is maintained as a git subtree of the standalone repo MoebiusX/otel-mcp-server. The standalone repo is the single source of truth.

Pull latest changes from upstream

cd /path/to/KrystalineX
git subtree pull --prefix=otel-mcp-server otel-upstream main --squash

Push monorepo changes back upstream

cd /path/to/KrystalineX
git subtree push --prefix=otel-mcp-server otel-upstream main

Initial setup (already done)

git remote add otel-upstream https://github.com/MoebiusX/otel-mcp-server.git
git subtree add --prefix=otel-mcp-server otel-upstream main --squash

Note: src/client.ts is currently monorepo-only. Push it upstream when ready.

License

Apache-2.0 — see LICENSE.

Available Tools

29 tools
anomalies_activeC

Get currently active anomalies detected by trace-based and amount-based detectors.

Inputs: type.

anomalies_baselinesA

Get current span duration baselines used for anomaly detection — mean, stdDev, p50, p95, p99 per operation.

Inputs: no named parameters.

backend_capabilitiesB

Report supported product versions and protocol feature availability per backend. Optionally filter by skill and classify a specific backend version into its support tier with a per-feature availability breakdown.

Inputs: skill, backend, version.

exchange_statusB

Get the KrystalineX exchange overall status: operational state, uptime, current observability posture, and last incident summary.

Inputs: no named parameters.

logs_labelsB

List all label names available in Loki.

Inputs: target.

logs_label_valuesA

Get all values for a Loki label (e.g. all app names, namespaces, components).

Inputs: label (required), target.

logs_queryB

Query logs from Loki using LogQL. Returns log lines matching the query.

Inputs: query (required), end, limit, start, +2 more.

logs_tail_contextA

Get recent logs correlated with a specific trace ID. Searches across all apps for log lines containing the trace ID.

Inputs: trace_id (required), limit, target.

metrics_alertsA

Get all Prometheus alerting rules and their current state (firing, pending, inactive).

Inputs: filter, target.

metrics_label_valuesB

Get all values for a given Prometheus label (e.g. list all jobs, instances, or status codes).

Inputs: label (required), target.

metrics_metadataA

Look up metric metadata — type, help text, and unit for a metric name.

Inputs: metric (required), target.

metrics_queryB

Execute an instant PromQL query and return current metric values.

Inputs: query (required), time, target.

metrics_query_rangeC

Execute a range PromQL query and return a time series.

Inputs: end (required), query (required), start (required), step, +1 more.

metrics_targetsA

List all Prometheus scrape targets and their health status.

Inputs: target.

recent_tradesB

Get the most recent anonymized public trades on KrystalineX. Useful for "is the market active?" questions. Trades carry no user identity.

Inputs: limit.

system_healthA

Get full system health — service status, uptime, performance metrics, active alerts.

Inputs: no named parameters.

system_topologyB

Get the live service dependency topology with health overlays from Jaeger and the application API.

Inputs: no named parameters.

total_volumeA

Get aggregate trading volume on the KrystalineX exchange (24h, weekly, all-time). Returns volume per asset and totals.

Inputs: no named parameters.

trace_getB

Get full trace detail by ID. Backend: Jaeger.

Inputs: trace_id (required).

traces_dependenciesC

Service dependency graph — which services call which. Backend: Jaeger.

Inputs: lookback.

traces_operationsC

List operations/endpoints for a service. Backend: Jaeger.

Inputs: service (required), lookback.

traces_servicesC

List services known to the traces backend. Backend: Jaeger.

Inputs: lookback.

transparency_metricsB

Get the full transparency metrics bundle: trace coverage, anomaly counts, MTTR, ZK proof generation rate, etc. Powers the public "Proof of Observability" dashboard.

Inputs: no named parameters.

verify_traceA

Look up the public-safe distributed trace for a given trade ID — service hops, timing, and verification result. Useful for "show me what happened to this trade".

Inputs: trace_id (required).

zk_proof_getB

Retrieve a ZK-SNARK proof for a specific trade. Returns the Groth16 proof, public signals, and verification key.

Inputs: trade_id (required).

zk_proof_verifyB

Verify a ZK-SNARK proof server-side. Returns whether the Groth16 proof is mathematically valid.

Inputs: trade_id (required).

zk_solvencyB

Get the latest solvency proof — proves total reserves ≥ liabilities without revealing individual balances.

Inputs: no named parameters.

zk_statsA

Get aggregate ZK proof statistics — total proofs generated, verification success rate, average proving time, circuit breakdown.

Inputs: no named parameters.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 29 tools

    Disambiguation4/5

    Most tools target a distinct backend+resource (metrics_query vs logs_query vs traces_search), and domain prefixes make routing obvious. There is mild overlap among the summary-style tools — system_health, exchange_status, and transparency_metrics all surface posture/alerts/uptime, and verify_trace reads like a verification verb but is actually a business-level trace lookup (contrast zk_proof_verify). These edge cases are clarified by descriptions, so misselection risk is low but not zero.

    Naming Consistency3/5

    The dominant pattern is snake_case with a domain prefix (metrics_*, traces_*, logs_*, zk_*, anomalies_*, system_*), which is readable and mostly predictable. However, six tools drop the prefix entirely (exchange_status, total_volume, recent_trades, transparency_metrics, verify_trace, backend_capabilities), and suffix styles mix verbs and nouns (traces_services vs traces_search, zk_solvency vs zk_proof_get), so the convention is not fully uniform.

    Tool Count3/5

    At 29 tools the surface is on the heavy side and exceeds the comfortable 3-15 band, with each domain contributing 4-6 tools. The breadth is justified by the server spanning four distinct backends (Prometheus, Jaeger, Loki, ZK) plus a business-exchange domain, so it is borderline rather than bloated. A few summary tools (exchange_status, transparency_metrics, system_health) could be consolidated.

    Completeness4/5

    Coverage is strong across every advertised domain: metrics (targets, instant/range queries, alerts, metadata, label values), traces (get, services, operations, dependencies, search), logs (query, label names/values, trace correlation), ZK (proof get/verify, solvency, stats), anomalies, and system topology/health. Minor gaps remain — there is no Prometheus label-name listing to mirror logs_labels, no span-level retrieval, and no ZK proof history/listing — but agents can work around these via the query tools.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables secure enterprise AI agents to access internal tools like GitHub, Gmail, Calendar, file systems, databases, and knowledge bases through the Model Context Protocol, with built-in security, audit, and observability.
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to query OriginChain's telemetry agent for log patterns, semantic incident analysis, distributed traces, service graphs, and read-only SQL queries over telemetry data.
      10
      4 npm
      MIT
    • F
      license
      Not graded
      quality
      C
      maintenance
      Enables secure, stateful AI agents and LLM tools to access enterprise databases, knowledge bases, live system metrics, and sandboxed code execution through the Model Context Protocol, with authentication, rate limiting, and safety guards.
      -