FABRIC Infrastructure Metrics MCP Server

A Model Context Protocol (MCP) server that lets LLM clients (Claude Desktop, Claude Code, VS Code Copilot, Chatbox, etc.) ask natural-language questions about the infrastructure metrics of the FABRIC Testbed — a nation-wide programmable network research infrastructure.

Key features

• Grounded by construction — every tool runs a fixed PromQL query authored in a FABRIC Grafana dashboard. The model never writes PromQL; it picks a tool, a variant, and parameters.

• 17 tools — 14 metric topics (CPU, memory, disk, filesystem, network, sockets, netstat, system, time, temperature, link traffic, link status, port traffic) plus 3 discovery tools.

• Two modes — a local stdio process using your own CILogon session, or one served deployment holding a Grafana service-account token.

• Friendly targets — amst means every node at that site, cern worker 2 resolves to cern-w2; the server maps them to real instances.

• Bounded responses — series, samples, and total points are capped and downsampled to fit an LLM context without losing the time window.

• Production-ready — CredMgr token verification, per-caller rate limiting, structured access logs, Prometheus metrics, and a startup credential probe.

Table of contents

Related MCP server: RHOAI Observability MCP

Quick install

Prerequisites: Python 3.11+, a FABRIC token from portal.fabric-testbed.net (Experiments → Manage Tokens), and — for local mode only — a browser for the one-time Grafana login.

# Remote mode: talk to the deployed server (no Grafana credential needed)
curl -fsSL https://raw.githubusercontent.com/fabric-testbed/fabric_metrics_mcp/main/install.sh | bash -s -- --remote

# Local mode: run the server yourself against your own Grafana session
curl -fsSL https://raw.githubusercontent.com/fabric-testbed/fabric_metrics_mcp/main/install.sh | bash -s -- --local

The installer:

1. Creates a virtual environment and installs the package.

2. Writes run_stdio.sh (local) or run_remote.sh (remote) with your paths baked in.

3. Prints the MCP client configuration to paste.

4. Runs a smoke test against the server.

Note: Remote mode needs nothing but a FABRIC token — the deployed server holds its own Grafana credential. Prefer it unless you are developing the server itself.

Manual setup: see Local mode setup or Remote mode setup.

MCP client configuration

Both helper scripts speak stdio, so any MCP client can launch them:

• run_stdio.sh — runs the server locally

• run_remote.sh — bridges stdio to the deployed HTTPS server, injecting your token

Important: keep the server name short. The model sees mcp__<server-name>__<tool>, and OpenAI-compatible stacks reject function names over 64 characters. fabric_metrics_list_param_options is already 33; a long server name silently drops the longest tools.

Claude Code CLI

claude mcp add fabric-metrics -- /path/to/run_remote.sh

Claude Desktop

{
  "mcpServers": {
    "fabric-metrics": {
      "command": "/path/to/run_remote.sh"
    }
  }
}

VS Code

{
  "servers": {
    "fabric-metrics": {
      "type": "stdio",
      "command": "/path/to/run_remote.sh"
    }
  }
}

Chatbox

{
  "name": "fabric-metrics",
  "command": "/path/to/run_remote.sh",
  "env": {
    "FABRIC_TOKEN_LOCATION": "/path/to/id_token.json",
    "FABRIC_MCP_URL": "https://<your-host>/mcp"
  }
}

Tools reference

Every topic tool takes variant (which measurement), the dashboard's parameters, and an optional time window. Call fabric_metrics_list_topics() to route a question.

Node metrics

Switch and link metrics

Discovery

Note: variant is an argument, never a tool name. There is no oom_killer tool — it is fabric_metrics_memory(variant="oom_killer", ...).

Authentication

There are two independent boundaries. The caller's token is never forwarded to Grafana.

Get a token: portal.fabric-testbed.net → Experiments → Manage Tokens, or fabric-cli tokens create. Identity tokens last about four hours.

Important: this server terminates authentication rather than proxying it — it queries Grafana with a credential broader than any caller's. FABRIC_VERIFY_TOKENS=1 is therefore mandatory in any reachable deployment; an unverified JWT payload is just base64.

Architecture

  "How busy is the CPU on cern-w5?"
        │
        ▼  LLM picks tool + variant + params
  fabric_metrics_cpu(variant="busy", node="cern-w5")
        │
        ▼  HTTPS + Authorization: Bearer <FABRIC token>
  ┌─────────────────────────────────────────────────┐
  │ nginx (TLS)  →  fabric-metrics-mcp              │
  │                   ├─ verify token (CredMgr)     │
  │                   ├─ resolve node → instance    │
  │                   └─ substitute into the panel's│
  │                      FIXED PromQL               │
  └─────────────────────────────────────────────────┘
        │
        ▼  Authorization: Bearer <Grafana service token>
  Grafana  →  datasource proxy  →  Mimir / Prometheus
        │
        ▼
  → 0.8 %   (grounded, real data)

• The catalog of dashboards, panels, and their queries ships inside the package, so the running server has no dependency on the repo layout or on Grafana's dashboard API.

• Responses are capped at 60 series and 3000 total samples; beyond that the step widens to fit while keeping the full window.

Repo layout

fabric_metrics_mcp/
  __main__.py          # entry point: transport, middleware, prompt registration
  config.py            # all configuration, read once from the environment
  registry.py          # topic -> variant -> panel, loaded from the shipped catalog
  query.py             # time parsing, substitution, Grafana calls, response budget
  tools.py             # one tool per topic + discovery tools
  session.py           # the SERVER's Grafana credential (token or cookies)
  system.md            # served as the `fabric-metrics-system` MCP prompt
  auth/resolver.py     # the CALLER's FABRIC token
  middleware/          # access log, rate limiting
  data/annotated/      # per-dashboard panels: fixed PromQL, units, legends
  data/groups/         # the topic -> variant taxonomy
deploy/
  central.env.example  # one config file for the served deployment
  local.env.example    # one config file for a local Docker run
deploy.sh              # mode selector + preflight, wraps docker compose
DEPLOY.md              # step-by-step deployment runbook
docker-compose.yml     # central: mcp-server + optional nginx (tls profile)
nginx/default.conf     # TLS termination, bearer-only /mcp, restricted /metrics
tests/                 # offline unit tests (no network, no credentials)
step2_build_catalog.py # rebuild: scrape dashboards
step3_annotate.py      # rebuild: authored descriptions
step3_5_group.py       # rebuild: topic/variant taxonomy
test_server.py         # end-to-end smoke test + LLM evaluation harness

Environment variables

Configuration is read once at startup. Deployment settings live in deploy/central.env (see Deploy with Docker Compose).

Deployment-only variables, consumed by docker-compose.yml rather than the app:

Note: the server also publishes its operating instructions as an MCP prompt named fabric-metrics-system.

Deploy with Docker Compose (Server Mode)

The full runbook — including network prerequisites and troubleshooting — is in DEPLOY.md. Summary:

Step 1: Confirm the Grafana credential works from this host

export GRAFANA_TOKEN='glsa_...'
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  https://infrastructure-metrics.fabric-testbed.net/grafana/api/user

200 means the token reaches Grafana. A 302 to /login means an auth proxy intercepts it — point FABRIC_METRICS_BASE_URL at Grafana's internal address instead.

Step 2: Place your TLS certificates

sudo mkdir -p /etc/ssl/fabric-metrics
# copy the certificate (FULL CHAIN, not the leaf alone) and key into it
sudo chmod 600 /etc/ssl/fabric-metrics/privkey.pem

Important: ask your CA for the full chain. Browsers fetch missing intermediates themselves, so a leaf-only certificate looks fine in a browser and fails every command-line client.

Step 3: Create the deploy/central.env file

cp deploy/central.env.example deploy/central.env
chmod 600 deploy/central.env

At minimum set:

GRAFANA_TOKEN=glsa_...
FABRIC_METRICS_BASE_URL=<whatever Step 1 proved>
PUBLIC_HOSTNAME=<your hostname>
NGINX_BIND=<your public IP>
SSL_CERT=/etc/ssl/fabric-metrics/fullchain.pem
SSL_KEY=/etc/ssl/fabric-metrics/privkey.pem
FORWARDED_ALLOW_IPS=172.32.240.10
RATE_LIMIT_TRUSTED_PROXIES=172.32.240.10/32

Step 4: Start without TLS and test through a tunnel

./deploy.sh central up -d --build
./deploy.sh central logs --tail 40

Look for Grafana credential OK: authenticated to ... as '<service-account>'. Then from your workstation:

ssh -N -L 8000:127.0.0.1:8000 <user>@<host>          # leave running
python test_server.py --http http://127.0.0.1:8000/mcp

This proves the app and the Grafana credential work with no certificate involved.

Step 5: Start the services with TLS

./deploy.sh central --profile tls up -d

Two containers run:

• fabric-metrics-mcp — the MCP server, published on loopback only

• fabric-metrics-nginx — TLS termination, reverse proxy to :8000

The preflight prints the certificate's subject and expiry, refuses a mismatched certificate/key pair, and warns if PUBLIC_HOSTNAME is absent from the SANs.

Step 6: Verify

./deploy.sh central --profile tls ps            # both "running", app "(healthy)"

dig +short <your-hostname>                      # must resolve to this host

curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 https://<your-host>/mcp
#   406 = up (the MCP endpoint rejects a plain GET). Never use -I here.

export FABRIC_TOKEN_LOCATION=$HOME/id_token.json
python test_server.py --http https://<your-host>/mcp

Configuration files

• deploy/central.env.example — every setting, one file

• docker-compose.yml — services; nothing hardcoded

• nginx/default.conf — TLS and proxy rules

• deploy.sh — mode selector, preflight, passes everything else to docker compose

NGINX requirements, if you front this with your own proxy:

• proxy_set_header Authorization $http_authorization — the caller's token must arrive untouched

• proxy_buffering off and a long proxy_read_timeout — MCP streams responses

• proxy_set_header X-Real-IP $remote_addr — $remote_addr overwrites; X-Forwarded-For appends and must never key a rate limit

• No auth_request on /mcp — it is bearer-token-only, verified by the app

Rebuilding the catalog

The shipped catalog is a snapshot of five Grafana dashboards. Rebuild it when FABRIC changes a dashboard:

pip install -e '.[build]'
python step2_build_catalog.py     # scrape dashboards      -> catalog/
python step3_annotate.py          # authored descriptions  -> data/annotated/
python step3_5_group.py           # topic/variant taxonomy -> data/groups/

• step2 extracts panels, queries, units, legends, and template variables for the five featured dashboards, keyed by UID.

• step3 holds hand-authored descriptions; these are the source of truth for routing and are what the model reads.

• step3_5 assigns every panel to exactly one (topic, variant) and fails loudly on duplicates or omissions.

Note: panel IDs are referenced by number, so a deleted panel breaks the rebuild with a KeyError rather than silently vanishing.

Local mode setup

Runs the server on your machine against your own Grafana session, so it sees exactly what your account sees. No service token.

Quick install: curl -fsSL .../install.sh | bash -s -- --local

Step 1: Create a Python virtual environment

python3 -m venv .venv && source .venv/bin/activate

Step 2: Install the package

pip install -e '.[local]'

Step 3: Capture your Grafana session

python get_grafana_session.py

Opens a browser, waits for you to complete CILogon, and saves grafana_state.json. It expires in hours to days.

Step 4: Get your FABRIC token

Download id_token.json from the portal, or fabric-cli tokens create.

Step 5: Configure the script

export FABRIC_TOKEN_LOCATION=$HOME/id_token.json

Step 6: Test

python test_server.py

Step 7: Configure your MCP client

Point it at run_stdio.sh — see MCP client configuration.

Remote mode setup

Talks to the deployed server over HTTPS. Nothing runs locally except a stdio bridge.

Quick install: curl -fsSL .../install.sh | bash -s -- --remote

Step 1: Install prerequisites

# node (for npx mcp-remote) and jq
brew install node jq        # or: apt install nodejs npm jq

Step 2: Set up the venv

Only needed to run test_server.py; MCP clients need just the bridge script.

python3 -m venv .venv && source .venv/bin/activate && pip install -e .

Step 3: Create your token

Download id_token.json from the portal, or fabric-cli tokens create.

Step 4: Get the helper script

run_remote.sh from this repo. It reads your token, checks it has not expired, and bridges stdio to HTTPS.

Step 5: Configure the script

export FABRIC_TOKEN_LOCATION=$HOME/id_token.json
export FABRIC_MCP_URL=https://<your-host>/mcp

Important: use an absolute path for FABRIC_TOKEN_LOCATION. A relative ./id_token.json resolves against whatever directory your MCP client happens to start in.

Step 6: Test

python test_server.py --http https://<your-host>/mcp

Step 7: Configure your MCP client

Point it at run_remote.sh — see MCP client configuration.

Local vs Remote — which to use?

Recommendation: use remote mode unless you are changing the server itself. It needs only a FABRIC token, and the session never expires out from under you.

Variants, parameters & results

Naming the target

node accepts what a person would say; the server resolves it. Pass the user's target as-is.

Head node is hn — one per site; there is no h1. Workers are w1…wN. Site codes are opaque: present them as-is rather than expanding them into institution names.

Time ranges

start / end take natural language, like Grafana's time picker: "past 5 minutes", "last 30 min", "now-1h", "2 hours ago", "today 6am", "2026-01-13 15:00".

• No time given → last 5 minutes. Gauge panels evaluate at end, so they return the current value regardless.

• step is auto-chosen for ~120 points, never finer than the 30 s scrape interval.

• mode (instant | range) overrides the panel's default query type.

Reading results

Each result carries panel, unit, description, resolved_queries, params_used, and series. Each series has a name from the dashboard's legend.

• Honour unit. A percent panel is already a percentage.

• A range series carries values, not timestamped pairs: values[i] is the sample at window.start + i * window.step, and null is a gap — not zero.

• truncated: true means series were dropped (>60). downsampled: true means the step widened to fit the budget; the window is unchanged.

• Zero series is a real answer, not a reason to invent one.

• error: "invalid_param_value" means a fixed-option parameter got a value it can never match. The message names the valid values.

Quick tool examples

CPU utilization on one node

{ "name": "fabric_metrics_cpu",
  "arguments": { "variant": "busy", "node": "amst-w1" } }

Compare memory across a whole site

{ "name": "fabric_metrics_memory",
  "arguments": { "variant": "ram_used", "node": "cern" } }

The three hottest nodes in the fleet

{ "name": "fabric_metrics_temperature",
  "arguments": { "variant": "hottest", "params": { "toplimit": "3" } } }

Routing hints for confusable requests

System prompt

The server publishes its operating instructions as an MCP prompt, so a client can load them without hardcoding anything:

@mcp.prompt(name="fabric-metrics-system")
def fabric_metrics_system_prompt():
    return SYSTEM_TEXT

Tool names inside it track FABRIC_METRICS_TOOL_PREFIX, so the served prompt never goes stale.

Logging

• Every HTTP request is logged with method, path, status, duration, a request id, and the caller's identity.

• Every tool call logs start, completion with duration, sanitised parameters, and errors.

• LOG_FORMAT=json emits one JSON object per line for ingestion.

• Tokens are never logged — every helper is redacted by construction.

{"ts": "2026-09-16T15:45:19+0000", "level": "INFO", "logger": "fabric_metrics.access",
 "msg": "POST /mcp -> 200 in 1.98ms", "request_id": "f4ea4c72f16f",
 "user_email": "user@example.edu", "client_ip": "203.0.113.7", "duration_ms": 1.98}

Important: with a shared Grafana service account, Grafana's own logs cannot tell you which person ran which query. These access logs are the only per-user record — ship them somewhere durable.

Testing

pip install -e '.[test]'
pytest                                   # offline unit tests, no network or credentials

The suite covers the pure layers where a wrong answer is easiest to hide: PromQL substitution (including the regex and $__all forms that must keep working), the parameter-injection guard, time-window parsing, the request timeout, the startup credential probe, and catalog loading.

End-to-end, against a real server:

python test_server.py                              # local stdio
python test_server.py --http https://<host>/mcp    # a deployment
python test_server.py --eval --limit 20            # drive an LLM over test_questions.md

--eval needs an OpenAI-compatible endpoint (FABRIC_LLM_BASE_URL, FABRIC_LLM_API_KEY, FABRIC_LLM_MODEL) and writes report.html grading which tool and variant the model chose.

Monitoring & Metrics (Server Mode Only)

  MCP server  ──/metrics──▶  Prometheus  ──▶  Grafana
   (:8000)                    (scrape)         (dashboards)

• Enabled by default in server mode; needs the [metrics] extra.

• /metrics is restricted to the monitoring network by nginx/default.conf.

Disabling metrics

METRICS_ENABLED=0

Production considerations

• Per-IP labels are off by default — one Prometheus series per source address is unbounded on a public endpoint. METRICS_CLIENT_IP_LABELS=1 opts in deliberately.

• Rate-limit state is per-process and in memory — fine for one container; a multi-replica deployment needs a shared store, and two replicas double the effective limit.

• The healthcheck is a TCP probe — it cannot detect an expired Grafana credential. Watch for Grafana credential NOT working in the logs.

Security notes

• The caller's token is never forwarded. This server queries Grafana with its own credential, so it terminates authentication rather than proxying it.

• FABRIC_VERIFY_TOKENS=1 is mandatory when reachable. Without CredMgr verification the gate is decorative — a JWT payload is base64 anyone can write.

• Server mode never falls back to a local token file. A served process must not answer someone else's request with the operator's credential.

• Parameters cannot escape their label matcher. Values carrying a quote or backslash are rejected before substitution; regex metacharacters stay legal because dashboards legitimately use them.

• Every Grafana request has a finite timeout. Without one a hung upstream pins a worker thread until the process restarts.

• There is no per-user authorization. Every authenticated FABRIC user sees all sites' metrics — the same access they already have by logging into the dashboards.

Rate limiting behind a proxy

The limiter's key is its bucket, so it may only come from inputs a caller cannot forge — anything a caller controls is something they can rotate for a fresh bucket per request. But it must still distinguish callers: behind a reverse proxy the socket peer is the proxy for every request, so keying on it alone puts everyone in one bucket and turns RATE_LIMIT into a service-wide cap.

RATE_LIMIT_TRUSTED_PROXIES resolves this by naming the proxies allowed to assert the real client address:

1. List only your reverse proxy, as a /32. Any host inside a wider range could forge the header.

2. Use X-Real-IP, which nginx sets from $remote_addr and therefore overwrites. Never key on X-Forwarded-For — nginx appends to whatever the client sent, so its left-most entry stays caller-controlled even on a trusted hop.

3. Leave it empty when clients reach the server directly.

Per-user limiting: what it would take

Keying on a verified JWT subject would give true per-user limits regardless of address. It is not enabled because:

1. The Starlette helper that reads claims at the edge has no verifier hook, so claims there are unverified.

2. An unverified claim is attacker input, not identity — a payload can be hand-written with no signing key.

3. Verification happens in the tool layer, after the limiter has already run.

4. Address keying at the edge is the safe default; the cost is that callers behind one NAT share a bucket.

License

MIT