# mflib Slice Metrics MCP Server

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io/)

A **[Model Context Protocol (MCP)](https://modelcontextprotocol.io/)** server that lets LLM clients ask natural-language questions about the nodes in a **[FABRIC](https://fabric-testbed.net) slice** — answered by the fixed PromQL queries of the **Node Exporter Full** dashboard already running on the slice's mflib measurement node.

### Key features

- **Grounded by construction** — every tool runs a **fixed PromQL query from the dashboard**. The model never writes PromQL; it picks a tool, a variant, and parameters.
- **15 tools** — 12 metric topics covering all 112 panels, plus 3 discovery tools.
- **Runs where the data is** — one container on the measurement node, beside Prometheus and Grafana. `git clone`, `docker compose up`.
- **No auth of its own** — it listens on loopback and the mflib portal's proxy authenticates callers, so there is no second token to issue or rotate.
- **Handles arbitrary node names** — slices are built by people, not a naming scheme. Names resolve case-insensitively, unique prefixes work, and `list_nodes()` exists because guessing is not possible.
- **Correct multi-node comparison** — most dashboard panels are written for one node, so querying several runs the panel once per node rather than averaging them into a meaningless number.

---

## Table of contents

| Getting started | Reference | Operations |
|:---|:---|:---|
| [Quick start](#quick-start) | [Tools reference](#tools-reference) | [Configuration](#configuration) |
| [MCP client configuration](#mcp-client-configuration) | [Node names](#node-names) | [Rebuilding the catalog](#rebuilding-the-catalog) |
| [How it works](#how-it-works) | [Reading results](#reading-results) | [Testing](#testing) |
| | [Repo layout](#repo-layout) | [Troubleshooting](#troubleshooting) |

---

## Quick start

> **Prerequisites:** an mflib measurement node with Prometheus and Grafana running, Docker, and the Compose v2 plugin.

```bash
git clone https://github.com/fabric-testbed/mflib_portal_mcp.git
cd mflib_portal_mcp
docker compose up -d --build
```

No configuration step: `.env` ships with the standard measurement-node values, and
the Prometheus datasource is discovered from Grafana at startup. `.env.example`
documents every setting if you need to change one.

```bash
docker compose logs --tail 20
```

Look for:

```
Grafana OK: http://fabric_prometheus-grafana:3000 is readable
Datasource discovered: 'Prometheus' (uid=...)
Starting mflib-mcp on http://0.0.0.0:8000/mcp
```

The container publishes on `127.0.0.1:8000` only. Point the portal's proxy at that port.

Confirm it is serving:

```bash
curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 http://127.0.0.1:8000/mcp
```

`406` is success — the MCP endpoint refuses a plain `GET`. Do not use `curl -I`; the endpoint does not answer `HEAD` and the request will appear to hang.

---

## How it works

```
  "Which node is using the most memory?"
        │
        ▼  LLM picks tool + variant + params
  mflib_memory(variant="ram_used", node=["Node1","Node2","Node3"])
        │
        ▼  loopback, behind the portal's proxy
  ┌──────────────────────────────────────────────┐
  │ mflib-mcp                                    │
  │   ├─ resolve names against live label values │
  │   ├─ substitute into the panel's FIXED PromQL│
  │   └─ run once per node (panels aggregate)    │
  └──────────────────────────────────────────────┘
        │
        ▼
  Grafana ──▶ datasource proxy ──▶ Prometheus   (both on the fabric_prometheus network)
        │
        ▼
  → Node2, 14.4 %   (grounded, real data)
```

The dashboard's panels, queries, units and legends are extracted once and **ship inside the package**, so the running server needs no dashboard API and cannot drift mid-flight.

---

## Tools reference

Every topic tool takes `variant` (which measurement), `node`, and an optional time window. Call `mflib_list_topics()` to route a question.

| Tool | Variants | Use for |
|:-----|:--|:------------|
| `mflib_cpu` | 8 | utilization, load average, core count |
| `mflib_memory` | 12 | RAM/swap used, cache, paging, **page faults**, **OOM kills** |
| `mflib_memory_internals` | 13 | slab, vmalloc, hugepages, writeback, NFS |
| `mflib_disk` | 11 | throughput, IOPS, latency, queue depth |
| `mflib_filesystem` | 8 | free space, inodes, **read-only / error state** |
| `mflib_network` | 19 | bandwidth, packets, errors, link state, ARP, conntrack |
| `mflib_sockets` | 5 | TCP/UDP socket counts |
| `mflib_netstat` | 11 | connections, retransmits, ICMP, TCP errors |
| `mflib_system` | 16 | uptime, processes, **PIDs/threads vs limits**, interrupts, systemd |
| `mflib_time` | 4 | clock sync / NTP drift |
| `mflib_hardware` | 3 | temperature, cooling devices, power supplies |
| `mflib_exporter` | 2 | node_exporter's own scrape health |

Discovery: `mflib_list_nodes()`, `mflib_list_topics()`, `mflib_list_param_options()`.

`variant` is an **argument**, never a tool name — there is no `ram_used` tool; it is `mflib_memory(variant="ram_used", …)`.

**Routing hints:**

| Question | Tool and variant |
|:---|:---|
| OOM kills, page faults | `memory` — *not* `memory_internals`, despite the name |
| slab, vmalloc, hugepages | `memory_internals` |
| "is a disk filling up" | `filesystem` variant `space_used_pct` |
| "how busy is the disk" | `disk` variant `io_util`; "how slow" → `wait_time` |
| process counts, PID/thread limits | `system` |
| "is monitoring working" | `exporter` — this is scrape health, not node health |

These are virtual machines, so `mflib_hardware` usually returns **no series**: VMs expose no physical sensors. That is a real answer, not a failure.

---

## Node names

The single thing that makes this server different from a fixed-fleet one.

Nodes belong to **one slice**, and whoever built it chose the names — `Node1`, `web-server`, `alice-vm-3`. There is no scheme to infer them from.

```python
mflib_list_nodes()
# {"count": 3, "nodes": ["Node1", "Node2", "Node3"]}
```

- Matching is **case-insensitive**, and a **unique prefix** resolves (`node1` → `Node1`).
- An **ambiguous** prefix is left alone rather than resolved to whichever node sorts first — answering about the wrong machine is worse than answering about none.
- An **unknown** name returns no data, which is not the same as the node being idle.
- Pass a **list** to compare machines: `node=["Node1","Node2"]`.

That last point matters more than it looks. Most panels on this dashboard are written for a single node — `CPU Busy` is `count()`/`avg()` with no `by (instance)` — so interpolating a regex that matches three nodes averages them into one number. In testing that produced **−198.998 %**. The server therefore runs the panel once per node and labels each series.

---

## Reading results

Each result carries `panel`, `unit`, `description`, `resolved_queries`, `params_used`, and `series`. Each series has a `name` from the dashboard's legend.

- Honour `unit`. A `percent` panel is already a percentage.
- A range series carries `values`, **not** timestamped pairs: `values[i]` is the sample at `window.start + i * window.step`, and `null` is a gap — not zero.
- `truncated: true` means series were dropped (>60). `downsampled: true` means the step widened to fit the response budget; the window is unchanged.
- Zero series is a real answer, not a reason to invent one.
- `job` defaults to `node` and should stay there. The measurement node also scrapes `docker`, `ping` and `prometheus`, which export entirely different metrics.

---

## MCP client configuration

Clients reach this server through the mflib portal's proxy, which supplies the bearer token. The portal's own instructions are authoritative; the shape is:

```json
{
  "mcpServers": {
    "mflib": {
      "type": "http",
      "url": "https://<portal-host>/<slice>/mcp",
      "headers": { "Authorization": "Bearer <token from the portal>" }
    }
  }
}
```

Keep the server name **short** (`mflib` is ideal). The model sees `mcp__<server-name>__<tool>`, and OpenAI-compatible stacks reject function names over 64 characters — a long name silently drops the longest tools, starting with the discovery ones.

---

## Configuration

All settings are environment variables, read once at startup. `.env` ships with
working defaults; `.env.example` documents each one in full.

| Var | Default | Purpose |
|-----|---------|---------|
| `GRAFANA_CONTAINER` | `fabric_prometheus-grafana` | Grafana's container name — usually the only value that differs between deployments |
| `GRAFANA_PORT` | `3000` | Grafana's port inside the network |
| `GRAFANA_BASE_URL` | *composed from the two above* | Full-URL override; needed only when Grafana is not reachable as `container:port` (e.g. through an SSH tunnel) |
| `GRAFANA_DS_UID` | *empty — discovered at startup* | Prometheus datasource UID. Leave empty: mflib's Grafana provisioning gives the datasource no fixed `uid:`, so it is regenerated whenever Grafana restarts. Set it only on a Grafana whose provisioning pins one |
| `GRAFANA_TOKEN` | *empty* | Only if this Grafana disables anonymous reads |
| `GRAFANA_VERIFY_TLS` | `1` | `0` only for a self-signed Grafana over a tunnel |
| `MCP_PORT` | `8000` | Host-side port, published on loopback |
| `MCP_PATH` | `/mcp` | URL path the endpoint is served at. Must match what the proxy **forwards**: use `/` if its `location /mcp` strips the prefix (the way mflib's `location /grafana/` does), `/mcp` if it passes the prefix through |
| `MONITORING_NETWORK` | `fabric_prometheus` | The compose network Grafana runs on |
| `MFLIB_TOOL_PREFIX` | `mflib_` | Prefix on tool names; `""` for bare names |
| `MFLIB_VERBOSE_TOOLS` | `1` | `0` for terse tool descriptions — smaller surfaces reduce malformed tool calls on small models |
| `LOG_LEVEL` / `LOG_FORMAT` | `INFO` / `text` | `json` for structured logs |
| `ACCESS_LOG_USER_HEADER` | *empty* | Header the proxy sets to identify callers, logged if named |

**On the bind address.** This server performs no authentication: the proxy does, then forwards to loopback. The container binds `0.0.0.0` internally, so the loopback guarantee is the **port publish** in `docker-compose.yml` (`127.0.0.1:8000:8000`). Changing that to `0.0.0.0` exposes the slice's metrics to anything that can reach the VM.

---

## Rebuilding the catalog

The shipped catalog is a snapshot of the dashboard. Rebuild it if the dashboard changes:

```bash
python step1_fetch_dashboard.py --url http://localhost:3000/grafana   # -> catalog/
python step2_annotate.py                                             # -> data/annotated/
python step3_group.py                                                # -> data/groups/
```

- **step 1** pulls panels, queries, units, legends and template variables over Grafana's HTTP API.
- **step 2** holds the authored descriptions. These are the source of truth for routing — they are what the model reads to choose a tool.
- **step 3** assigns every panel to exactly one `(topic, variant)` and **fails** on a duplicate or an omission, so a new panel cannot be silently unreachable.

---

## Testing

```bash
pip install -e '.[test]'
pytest                      # 416 offline tests; no network, no credentials
```

They cover the layers where a wrong answer is easiest to hide: PromQL substitution, the parameter-injection guard, node-name resolution (including what it refuses to guess), the per-node fan-out, required-parameter rules, time parsing, request timeouts, catalog loading, datasource-UID handling, and the endpoint's mount path.

End-to-end, against a real measurement node. The published image already carries
every dependency, so this needs nothing installed on the node:

```bash
docker run --rm --network fabric_prometheus -v "$PWD:/src" -w /src \
  mflib-mcp:latest python test_server.py --http http://fabric_prometheus-mflib_mcp:8000/mcp
```

It performs a real MCP handshake, lists the tools, then runs live queries — node
names, CPU, RAM, filesystem, network, uptime and a 30-minute range query. Add `-v`
to print the values rather than series counts. It exits non-zero on failure.

Or directly, with the package installed:

```bash
python test_server.py                                 # stdio
python test_server.py --http http://127.0.0.1:8000/mcp
python test_server.py --eval --limit 20               # drive an LLM over test questions
```

`--eval` needs an OpenAI-compatible endpoint (`FABRIC_LLM_BASE_URL`, `FABRIC_LLM_API_KEY`, `FABRIC_LLM_MODEL`) and writes `report.html` grading which tool and variant the model chose.

---

## Repo layout

```
mflib_mcp/
  __main__.py          # entry point: transport, middleware, prompt registration
  config.py            # all configuration, read once from the environment
  registry.py          # topic -> variant -> panel, loaded from the shipped catalog
  query.py             # time parsing, substitution, fan-out, response budget
  tools.py             # one tool per topic + discovery tools
  session.py           # how this server reaches Grafana
  system.md            # served as the `mflib-system` MCP prompt
  middleware/          # access log
  data/annotated/      # panels: fixed PromQL, units, legends, descriptions
  data/groups/         # the topic -> variant taxonomy
step1_fetch_dashboard.py   # rebuild: pull the dashboard over Grafana's API
step2_annotate.py          # rebuild: authored descriptions
step3_group.py             # rebuild: topic/variant taxonomy
docker-compose.yml         # one container, loopback-published
test_server.py             # end-to-end smoke test + LLM evaluation harness
tests/                     # offline unit tests
.env                       # standard measurement-node values; no setup step needed
.env.example               # every setting, documented
integration/               # notes for merging into an existing measurement-node stack
```

---

## Troubleshooting

| Symptom | Cause |
|:---|:---|
| Startup says `Grafana NOT reachable` | `GRAFANA_BASE_URL` is wrong, or this container is not on Grafana's compose network. Check `docker network ls` and `MONITORING_NETWORK` |
| Every tool returns empty | Prometheus is not scraping the slice, or `GRAFANA_DS_UID` was pinned to a stale value. Check `mflib_list_nodes()` first — empty there means the datasource, not the tools. Leaving `GRAFANA_DS_UID` empty lets the server discover and re-resolve it |
| Every request 404s through the proxy | `MCP_PATH` does not match what the proxy forwards. A `location /mcp` that rewrites the prefix away delivers `/`; set `MCP_PATH=/` |
| `list_nodes` returns nothing | Prometheus has no `node_uname_info`; node_exporter is not running on the slice nodes |
| A node name returns no data | ambiguous or unknown, so it was passed through unchanged. Call `mflib_list_nodes()` for the exact spelling |
| `mflib_hardware` returns no series | expected on VMs — no physical sensors |
| Comparing nodes gives one odd number | a panel that aggregates was given a regex. The fan-out prevents this; if you see it, the `node` variable became `multi` upstream |
| `curl` hangs against `/mcp` | `-I`/HEAD is unanswered. Use a GET with `--max-time` |
| Tools missing in a client | the name is too long once the client's `mcp__<server>__` prefix is added. Shorten the server name |
| `400 Request Header Or Cookie Too Large` | the proxy's header buffers are too small for the portal's token. On nginx: `large_client_header_buffers 4 32k` |

```bash
docker compose logs --tail 40                 # startup probe + per-request log
docker compose exec mflib-mcp env | grep GRAFANA
```

---

## License

[MIT](./LICENSE)