# grafana-unified-mcp

One MCP server in front of many Grafana instances. Every tool from the standard
Grafana MCP server, plus one extra argument — `instance` — that says which
Grafana to run it against.

```
query_prometheus(instance="tenant-a", expr="up", datasourceUid="...")
search_dashboards(instance="tenant-b", query="login latency")
```

## Why this exists

Upstream [`grafana/mcp-grafana`](https://github.com/grafana/mcp-grafana) binds
`GRAFANA_URL` **once, at process start**. It reads
`X-Grafana-Service-Account-Token` per request, but the URL is fixed — and the
header that used to override it is now explicitly inert. From upstream
`validate_url.go`:

> `Deprecated: X-Grafana-URL no longer configures the Grafana client. This
> middleware is retained temporarily to preserve malformed-header handling.`

So one `mcp-grafana` process can only ever talk to one Grafana. Ten Grafanas
means ten servers, ten entries in every client config, and ten sets of tools
with identical names for the model to disambiguate.

This server fixes that by running one upstream child process per instance and
routing each call to the right one based on the `instance` argument. Tools are
discovered from the real binary at runtime, so you get whatever upstream
exposes — currently 65 tools — with no per-tool code here and nothing to update
when upstream adds more.

## How it works

```
                                        ┌──────────────────────────────────┐
  Claude Code / routines /              │  grafana-unified-mcp             │
  cloud sessions                        │                                  │
        │                               │  ┌────────────────────────────┐  │
        │  streamable-HTTP              │  │ bearer auth                │  │
        │  Authorization: Bearer …      │  │  → Principal(instances,    │  │
        ├──────────────────────────────►│  │      read-only|read-write) │  │
        │                               │  └────────────┬───────────────┘  │
        │                               │               │                  │
        │                               │  ┌────────────▼───────────────┐  │
        │                               │  │ catalog: inject `instance`  │  │
        │                               │  │ filter by caller's grant   │  │
        │                               │  └────────────┬───────────────┘  │
        │                               │               │ route on         │
        │                               │               │ instance=…       │
        │                               │  ┌────────────▼───────────────┐  │
        │                               │  │ child pool (lazy, reaped)  │  │
        │                               │  └──┬──────────┬──────────┬───┘  │
        └───────────────────────────────┴─────┼──────────┼──────────┼──────┘
                                              │ stdio    │ stdio    │ stdio
                                        ┌─────▼────┐ ┌───▼──────┐ ┌▼─────────┐
                                        │mcp-grafana│ │mcp-grafana│ │mcp-grafana│
                                        │ tenant-a │ │ tenant-b │ │   …      │
                                        └─────┬────┘ └───┬──────┘ └┬─────────┘
                                              ▼          ▼         ▼
                                          tenant-a    tenant-b   …Grafana
```

Children start on first use, stay warm, get reaped when idle
(`--idle-timeout`, default 15 min), and are respawned transparently if they die.
An unreachable Grafana degrades only its own instance.

## How to install

Docker is the supported path. The image builds the upstream `mcp-grafana` binary
itself, so the host needs no Go toolchain and no Python — just Docker.

### 1. Ports — what you actually need to open

**One port, however many Grafana instances you configure.** This is the part
people over-plan: the per-instance `mcp-grafana` children are *stdio*
subprocesses talking over pipes, so they consume no ports and need nothing
published. Ten Grafanas and one Grafana look identical from the network.

| Direction | Port | Purpose |
|---|---|---|
| Inbound | **21000** (container), one host port from your range | The MCP endpoint plus `/healthz` |
| Outbound | 443 → each Grafana host | Tool calls |
| Outbound | 443 → AWS Secrets Manager / SSM | Only if you read config from AWS |

Nothing ever connects *inward* from Grafana. There is no second listener, no
per-instance port, and no port-range requirement.

The container listens on 21000 internally. Map whichever host port you want from
your 21000–21999 range:

```bash
HOST_PORT=21000    # ELB target group points here
```

### 2. Build and run

```bash
git clone https://github.com/robert-sinclair/grafana-unified-mcp.git
cd grafana-unified-mcp

# Pin the upstream ref — it decides which tool catalogue gets republished.
export MCP_GRAFANA_REF=main
export HOST_PORT=21000
export CONFIG_DIR=/opt/grafana-unified-mcp/config
export PUBLIC_URL=https://grafana-mcp.example.com   # your ELB's URL

docker compose up -d --build
curl -s localhost:${HOST_PORT}/healthz | jq
```

The build takes a few minutes the first time — it compiles `mcp-grafana` from
source and then **verifies the binary publishes its tools before the image is
accepted**, so a broken upstream build fails here rather than at deploy.

Plain Docker, without compose:

```bash
docker build --build-arg MCP_GRAFANA_REF=main -t grafana-unified-mcp:local .

docker run -d --name grafana-unified-mcp --init --restart unless-stopped \
  -p 21000:21000 \
  -v /opt/grafana-unified-mcp/config:/etc/grafana-unified-mcp:ro \
  -e MCP_PUBLIC_URL=https://grafana-mcp.example.com \
  grafana-unified-mcp:local
```

`--init` matters: this process supervises one child per Grafana instance, so PID
1 has to reap them. Compose sets `init: true` for you.

### 3. Config, and the one permission that bites

Render `endpoints.json` and `auth.json` into `CONFIG_DIR` on the host and mount
them read-only. Do **not** copy them into the image — the endpoints document
holds a service-account token per Grafana, and an image layer is readable by
anyone who can pull it. The `.dockerignore` excludes them so this cannot happen
by accident.

**The container runs as UID 10001, and a bind mount keeps the host's ownership.**
Root-owned `0640` files — exactly what an Ansible `template` task produces by
default — are unreadable inside the container and the server exits at startup.
So have Ansible finish with:

```yaml
- name: Render unified MCP config
  ansible.builtin.template:
    src: "{{ item }}.j2"
    dest: "/opt/grafana-unified-mcp/config/{{ item }}"
    owner: "10001"          # matches the container user
    group: "10001"
    mode: "0640"            # readable in-container, not world-readable on host
  loop: [endpoints.json, auth.json]
  no_log: true              # the rendered files contain Grafana tokens
  notify: restart grafana-unified-mcp
```

UID 10001 maps to no real account on the host, so the files stay root-only there.

You do not need to restart to add or remove an instance — both documents are
re-read on an interval (`--config-refresh-seconds`, default 300). A restart is
only needed for changes to flags or the image.

### 4. Wire up the ELB

Listener **443 (HTTPS)** → target group **HTTP** on your host port. TLS
terminates at the load balancer, so the container serves plain HTTP and there is
no cert to mount.

Four settings are not the defaults, and each one fails in a way that looks like
something else:

| Setting | Value | Why |
|---|---|---|
| TG health check path | **`/healthz`** | The default `/` returns 404 and every target is marked unhealthy. `/healthz` needs no token and never touches Grafana, so it reports *this* process's health rather than a Grafana outage. |
| Idle timeout | **≥ 300s** | Streamable-HTTP holds a long-lived SSE stream. The ALB default of 60s cuts it, which surfaces as random client disconnects rather than an error. Keep it above `--call-timeout` (default 120s) too. |
| `MCP_PUBLIC_URL` | your ELB URL | The SDK validates the `Host` header for DNS-rebinding protection. The ELB forwards the client's public hostname, so without this **every** request to `/mcp` returns `421 Misdirected Request` with `Invalid Host header`. Health checks are unaffected — `/healthz` sits outside that middleware. |
| Security group | ELB's SG only | The container publishes on all host interfaces, so the SG is the real boundary — not `127.0.0.1`. |

Health-check specifics: protocol HTTP, path `/healthz`, port "traffic port",
success codes 200.

**If you run more than one target**, set `MCP_STATELESS=true`. MCP session state
is in-memory, so with two containers a client's follow-up request can land on the
one that has never heard of its session. ALB cookie stickiness is not a reliable
fix, because MCP clients are not browsers and need not send cookies. One target,
or stateless — those are the two correct configurations.

Also consider a short target-group deregistration delay: on redeploy, in-flight
SSE streams are cut, and clients reconnect.

### Running it without Docker

The container is the recommended path, but the server is a normal Python
package. It needs Python 3.11+ and the upstream binary on `PATH`:

```bash
deploy/install-mcp-grafana.sh /usr/local/bin   # needs Go 1.21+, GOTOOLCHAIN=auto does the rest
python3 -m venv /opt/grafana-unified-mcp/.venv
/opt/grafana-unified-mcp/.venv/bin/pip install 'grafana-unified-mcp[aws] @ .'
```

`deploy/` also carries a hardened systemd unit and an nginx example if you would
rather not use containers at all. Point at an existing binary with
`MCP_GRAFANA_BINARY=/path/to/mcp-grafana`.

## Configure

### Endpoints

Exactly the shape you'd expect — instance name to the upstream env vars:

```json
{
  "tenant-a": {
    "GRAFANA_URL": "https://tenant-a.example.cloud/grafana",
    "GRAFANA_SERVICE_ACCOUNT_TOKEN": "glsa_…"
  },
  "tenant-b": {
    "GRAFANA_URL": "https://tenant-b.example.cloud/grafana",
    "GRAFANA_SERVICE_ACCOUNT_TOKEN": "glsa_…",
    "description": "Tenant B production"
  }
}
```

Optional per-instance keys: `GRAFANA_ORG_ID`, `GRAFANA_USERNAME` /
`GRAFANA_PASSWORD`, `description`, `extra_env`, `extra_args`. To keep secrets
out of the document itself, use `GRAFANA_SERVICE_ACCOUNT_TOKEN_ENV` (read from
this process's environment) or `GRAFANA_SERVICE_ACCOUNT_TOKEN_FILE` (a path the
child reads).

### Auth

```json
{
  "clients": [
    {
      "name": "claude-routines",
      "token_sha256": "3f786850e387550fdab836ed7e6dc881de23001b…",
      "instances": ["tenant-a", "tenant-b"],
      "scope": "read-only"
    },
    {
      "name": "platform-oncall",
      "token_sha256": "…",
      "instances": ["*"],
      "scope": "read-write"
    }
  ]
}
```

Mint a token and its hash:

```bash
grafana-unified-mcp --hash-token          # generates one
grafana-unified-mcp --hash-token 'my-existing-token'
```

Give `token` to the client; put `token_sha256` in the document. Tokens are
compared by digest under `hmac.compare_digest`, and every client is checked on
every attempt so match position doesn't leak through timing.

Two things are enforced per caller:

- **`instances`** — the `instance` enum a caller sees is narrowed to its grant,
  and a call naming an instance outside it is refused with the same message as
  a nonexistent one, so a token can't enumerate what it can't reach.
- **`scope`** — `read-only` callers never even *see* mutating tools. The split
  comes from upstream's own `readOnlyHint` annotation (49 of 65 tools are
  read-only today), not a list maintained here, so tools added upstream are
  classified without a code change. Anything unannotated is treated as
  *not* read-only.

For belt-and-braces, add `--child-arg=--disable-write` to strip write tools at
the source for every caller.

### Running without authentication

`--auth-mode none` serves every caller that can reach the port, **read-only**.
There is no identity to scope instances by, so all configured instances stay
readable — but nothing is writable, because an open port should not be able to
rewrite a dashboard or delete a snapshot. That's enforced at three layers:

1. the published catalogue omits every mutating tool;
2. the authorization check refuses them even if a client names one directly;
3. children are started with `--disable-write`, so upstream refuses them too.

The third layer is what makes it more than a filter. Upstream swaps
`grafana_api_request` for a separate GET-only registration — no `body`
parameter, `method` narrowed to `GET`, non-GET rejected at runtime — so even a
bug in layers 1 and 2 could not turn into a write.

stdio is different: the local caller already holds the endpoints document and
every token in it, so restricting them would be theatre. stdio gets full access.

If you need writes over HTTP, use bearer tokens with a `read-write` client
rather than an open port.

### Where config comes from

Any of these, for both `--endpoints` and `--auth`:

| Source | Example |
|---|---|
| File | `/etc/grafana-unified-mcp/endpoints.json` |
| Inline env var | `env:GRAFANA_ENDPOINTS_JSON` |
| AWS Secrets Manager | `aws-secrets:prod/grafana/endpoints?region=us-west-2` |
| AWS SSM Parameter Store | `aws-ssm:/prod/grafana/endpoints?region=us-west-2` |

Both documents are re-read every `--config-refresh-seconds` (default 300). A
failed refresh logs and keeps the last good value, so a transient AWS error or a
half-written file can't take the server down. Adding an instance needs no
restart; removing one stops its child.

Validate before starting:

```bash
grafana-unified-mcp --endpoints … --auth … --check-config
```

## Run

In the container everything is already wired (see [How to install](#how-to-install)).
Running it directly:

```bash
# local, over stdio (no auth — the local caller already holds the config)
grafana-unified-mcp --endpoints ./examples/endpoints.json

# streamable-HTTP, reading config from AWS
grafana-unified-mcp \
  --transport streamable-http \
  --address 0.0.0.0:21000 \
  --endpoints aws-secrets:prod/grafana/endpoints?region=us-west-2 \
  --auth      aws-secrets:prod/grafana/mcp-auth?region=us-west-2 \
  --public-url https://grafana-mcp.example.com
```

Every transport flag has an environment fallback, so the image needs no
entrypoint script: `MCP_TRANSPORT`, `MCP_ADDRESS`, `MCP_PUBLIC_URL`,
`MCP_ALLOWED_HOSTS` (comma-separated), `MCP_AUTH_MODE`, `MCP_STATELESS`, plus
`GRAFANA_ENDPOINTS_SOURCE` and `MCP_AUTH_SOURCE`. Run `--help` for the rest.

`GET /healthz` reports process health, live children, and catalog state without
touching Grafana — which is why it is the right target-group health check.

## Connect a client

`.mcp.json`, for local stdio use:

```json
{
  "mcpServers": {
    "grafana": {
      "command": "/opt/grafana-unified-mcp/.venv/bin/grafana-unified-mcp",
      "args": ["--endpoints", "/etc/grafana-unified-mcp/endpoints.json"]
    }
  }
}
```

For the deployed server — including Claude Code routines and cloud sessions,
which is the case the bearer tokens exist for:

```json
{
  "mcpServers": {
    "grafana": {
      "type": "http",
      "url": "https://grafana-mcp.example.com/mcp",
      "headers": {
        "Authorization": "Bearer ${GRAFANA_UNIFIED_MCP_TOKEN}"
      }
    }
  }
}
```

Set `GRAFANA_UNIFIED_MCP_TOKEN` in the environment the session runs in — for
Claude Code on the web, that's the environment's variables, so scheduled
routines and cloud sessions pick it up without the secret living in the repo.
Give routines a `read-only` client; keep `read-write` for humans.

## Deploying without containers

[`deploy/`](deploy/) carries a hardened systemd unit (`ProtectSystem=strict`,
`PrivateTmp`, `NoNewPrivileges`, empty `CapabilityBoundingSet`), an installer,
and an nginx example:

```bash
sudo deploy/install.sh                       # user, dirs, venv, unit file
sudo systemctl edit grafana-unified-mcp      # set the source URIs / region
sudo systemctl enable --now grafana-unified-mcp
```

The nginx example disables response buffering, which is required for SSE
streaming — the same reason the ELB needs a raised idle timeout.

## Using it

Point the model at `list_grafana_instances` first:

```
list_grafana_instances()
→ { "instances": [ {"name": "tenant-a", "url": "…", "connection": "live"}, … ],
    "routing_argument": "instance",
    "access": { "client": "claude-routines", "scope": "read-only" } }
```

Then every other tool takes that name:

```
search_dashboards(instance="tenant-a", query="latency")
```

Pass `check_health=true` to also probe each Grafana — slower, since it opens a
connection to every instance.

### One naming wrinkle

Upstream's `grafana_api_request` already has a required parameter called
`endpoint` (the API path). Injecting a routing argument by that name would
silently shadow it, which is why the routing argument is `instance` by default.
If you rename it with `--routing-param endpoint`, that tool's own parameter is
automatically republished as `api_path` and mapped back on the way through — no
tool is ever broken by the collision, whatever you choose.

## Development

```bash
uv venv && uv pip install -e '.[dev,aws]'
uv run pytest                       # unit + integration
```

The integration tests drive a real `mcp-grafana` child against an unreachable
Grafana: enough to prove catalog discovery, `instance` injection and stripping,
routing, and auth filtering, without needing live credentials. Set
`MCP_GRAFANA_BINARY` to point at the binary, or they skip.

## Roadmap

- **OAuth 2.1** — the auth layer is already an interface, and the SDK already
  takes an OAuth provider alongside the token verifier. Filling in
  `OAuth2Provider.verify_token` is the whole job; `auth/oauth.py` documents the
  three steps. Map IdP groups onto the existing `grafana:read` /
  `grafana:write` / `instance:<name>` scopes and every authorization check keeps
  working unchanged.
- **Fan-out** — `instance: "*"` to run one read-only query across every
  instance and merge results. Useful for "which of these is alerting?"; left out
  for now because result merging deserves its own design.