# Kong AI Gateway Diagnostics

An MCP server that lets an LLM agent investigate a Kong Gateway through fourteen
read-only, bounded, schema-validated tools, with a deterministic diagnostic engine
producing the facts, a bounded agent loop producing the explanation, and an evaluation
harness measuring whether it works.

```
User:   Why is authentication failing on /payments?

Agent:  find_route_by_path({"path":"/payments"})        ->  route:payments-prod
Agent:  diagnose_route({"route":"route:payments-prod"})   ->  NO_AUTH_PLUGIN (HIGH), CONFIRMED

Agent:  Authentication is not failing on /payments because no authentication plugin
        is attached or enabled for that route at all. Kong is passing every request
        through without requiring credentials. [evidence, next steps, limitations]

        3 steps · 2 tool calls · 12,977 tokens · $0.0046 · 5.0 s
```

Kong is the source of truth. The model selects tools, sequences the investigation and
writes the explanation; it never produces a fact.

**Status:** a working reference implementation, verified against a local Kong 3.9.3 and
the live Gemini API. It is not deployed anywhere and has not been used in production.

---

## Problem

Debugging an API gateway is a sequence of narrow questions against a large, structured
configuration: which route serves this path, which service does it point to, which
plugins actually apply once you account for scope precedence, is the upstream healthy,
does this consumer hold the credential the route requires. Each question is a specific
Admin API call. The failure modes are not exotic -- a disabled plugin that still shows in
the UI, a route attached to the wrong service, a `503` that means "wrong address" rather
than "backend down" -- but they are easy to miss when eyeballing JSON, and the obvious
first hypothesis is often wrong.

An LLM is good at sequencing an investigation from a vague description. It is bad at
being trusted with infrastructure state. This project is about getting the first
property without paying for the second.

## Why MCP?

The Model Context Protocol gives the model a fixed vocabulary of operations, each with a
schema, a description of when to use it, and a bounded result -- instead of an HTTP
client and a URL. That makes the entire attack and blast surface enumerable (see
[`src/mcp/registry.ts`](src/mcp/registry.ts)), lets the same tools serve any MCP client
and this repository's own agent from one definition, and puts a protocol-enforced
validation boundary between the model and the gateway. There is deliberately no
`execute_kong_api_call` tool and never will be
([decisions.md](docs/decisions.md#2-why-not-direct-api-calls-from-the-agent)).

## Architecture

```mermaid
flowchart LR
    Client["AI client<br/>(Claude Desktop, IDE, or npm run agent)"]
    subgraph S["kong-ai-gateway-mcp"]
        direction TB
        MCP["MCP tools (14, read-only)"]
        Ctx["Context management<br/>references · summaries · limits"]
        Diag["Diagnostic engine<br/>evidence collector · pure rules"]
        Kong["Kong client<br/>typed · GET only · bounded"]
        Safety["Safety: permissions · audit"]
        Tel["Telemetry: logs · metrics · cost"]
        MCP --> Ctx --> Diag --> Kong
        MCP -.-> Safety
        MCP -.-> Tel
    end
    subgraph A["Agent + evaluation (needs an LLM key)"]
        Loop["Bounded agent loop"] --> Prov["LLM provider interface"] --> Gem["Gemini provider"]
        Eval["Evaluation harness<br/>28 cases · deterministic scorer"] --> Loop
    end
    Client -- MCP / stdio --> MCP
    Loop -- in-process --> MCP
    Kong -- Admin API --> KG[(Kong 3.9<br/>+ Postgres)]
    Gem --> API[(Gemini API)]
```

The tool layer runs without any LLM. Full component map, data flow and responsibilities
in [docs/architecture.md](docs/architecture.md).

## Features

- **MCP tooling** -- 14 tools on the official TypeScript SDK v2 (`@modelcontextprotocol/server`):
  `find_route_by_path`, `get_route`, `get_service`, `list_services`,
  `list_routes_for_service`, `list_plugins_for_route`, `list_global_plugins`,
  `get_consumer`, `list_consumers`, `list_upstreams`, `check_upstream_health`,
  `diff_config`, `diagnose_route`, `explain_auth_failure`. Every description states what
  the tool does, when to use it, what it returns, and its limits.
- **Kong integration** -- a typed client with explicit methods only, Zod-validated
  responses, a stable error taxonomy, response and page-count ceilings, and credential
  values stripped at the boundary. Kong is treated as authoritative.
- **Deterministic diagnostics** -- eleven pure rules over an evidence bundle that can
  only be filled from Kong. Absence is asserted only when every relevant scope was read.
  Runs in ~64 ms with no model. Detects all ten seeded defects and reports the healthy
  control as clean.
- **Agentic investigation** -- a loop this project owns (not the SDK's), with a step
  limit, identical-call detection, a cumulative context budget, and a halt path that
  produces an explicit "diagnosis incomplete" rather than a guess.
- **Context management** -- references instead of embedded entities, allowlisted plugin
  config projection, announced truncation, cursor pagination written for a model,
  progressive and conditional collection. Measured: a single-route diagnosis is 11% of
  the raw configuration dump.
- **Safety controls** -- no write method exists in `src/`; write tools are gated at
  registration; the server refuses to start with `ENABLE_WRITE_TOOLS=true` because none
  are implemented; secrets are stopped at each boundary they cross, with a test for each.
- **Evaluation** -- 28 cases, three difficulties, varied phrasings, false-positive
  controls, a deterministic scorer with no LLM judge, quota-aware and resumable runs,
  JSON and Markdown reports.
- **Observability** -- structured JSON logs (stderr only; stdout is the protocol), per-call
  telemetry with three correlation ids, a separate audit log, nearest-rank percentiles,
  provider-reported tokens, and cost that says "unavailable" rather than guessing.
- **Provider abstraction** -- one-method `LlmProvider` interface; `@google/genai` imported
  in exactly one file; two Gemini-specific protocol requirements discovered during
  development were absorbed without touching the agent, tools or evaluation.

## Demo

Actual output of `npm run agent -- --trace "Why is authentication failing on /payments?"`
against the seeded gateway and `gemini-3.5-flash-lite` (run captured on 2026-09-12):

```
Investigation trace:

  Step 1  (llm 1602ms, finish=tool_calls)
    -> find_route_by_path({"path":"/payments"})  ok 30ms 209B

  Step 2  (llm 1317ms, finish=tool_calls)
    -> diagnose_route({"route":"route:payments-prod"})  ok 64ms 3162B

  Step 3  (llm 1971ms, finish=stop)

DIAGNOSIS
**Diagnosis:**
Authentication is not failing on `/payments` because **no authentication plugin is
attached or enabled** for that route at all. Kong is currently passing all requests to
`/payments` through without requiring any credentials.

**Evidence:**
- The route `payments-prod` (serving `/payments`) has only a `cors` plugin attached at
  the route scope.
- There are no service-level plugins on `payments-api`.
- The global plugins active on the gateway are `request-size-limiting` and
  `correlation-id`.
- None of these are authentication plugins (`key-auth`, `jwt`, `basic-auth`, etc.).

**What to do next:**
1. Confirm with your team whether the `/payments` endpoint is intentionally public.
2. If authentication *should* be required, configure and enable an appropriate
   authentication plugin (such as `key-auth` or `jwt`) on the route or its backing
   service.

**Limitations:**
This diagnosis reflects gateway configuration only. If clients are receiving 401 or 403
responses when calling `/payments`, those responses are coming from the upstream backend
service (`http://payments-upstream:8080/v1`), not from Kong itself.

RUN METRICS
  steps                3 / 10
  tool calls           2 (0 error(s))
  repeated calls       0
  tools used           find_route_by_path, diagnose_route
  tool latency avg/p95 46.5ms / 64ms
  llm calls            3 (avg 1630ms)
  tokens               12677 in / 300 out / 12977 total
  estimated cost       $0.004553
  context used         3371 / 50000 chars
  wall clock           4993ms
```

The question's premise is wrong -- nothing is _rejecting_ requests -- and the agent
corrects it from evidence rather than accepting it. The `diagnose_route` result it
reasoned over is the same structured object an MCP client would receive:

```json
{
  "confidence": "CONFIRMED",
  "summary": "No enabled authentication plugin applies to route route:payments-prod. ...",
  "findingCount": 2,
  "findings": [
    {
      "code": "NO_AUTH_PLUGIN",
      "category": "AUTHENTICATION",
      "severity": "HIGH",
      "entityType": "route",
      "entityRef": "route:payments-prod",
      "observedState": "No enabled authentication plugin applies to route route:payments-prod. No authentication plugin is attached at route, service or global scope.",
      "expectedState": "A route serving protected resources should have an enabled authentication plugin (key-auth, jwt, basic-auth, oauth2, ...) at route, service or global scope.",
      "evidence": [
        "Route-scoped plugins: cors.",
        "Service-scoped plugins: none.",
        "Global plugins: request-size-limiting, correlation-id."
      ],
      "recommendedNextStep": "Kong forwards every request to /payments without authenticating it, ..."
    }
  ],
  "observed": { "route": { "ref": "route:payments-prod", ... }, "effectivePlugins": [ ... ] },
  "gaps": [],
  "suggestedNextSteps": [ ... ]
}
```

## Installation

Requirements: Node.js >= 20, npm, Docker with Compose v2.

```
git clone <this repository>
cd kong-ai-gateway-mcp
npm install
cp .env.example .env        # defaults work for the local Kong; add GEMINI_API_KEY for the agent
```

Verify the toolchain without Docker or an API key:

```
npm run check               # lint + typecheck + 140 unit tests
```

## Running Kong

```
npm run kong:up             # docker compose up -d  (Postgres, migrations, Kong 3.9)
npm run seed                # 12 services, 12 routes, 4 upstreams, 7 consumers, 20 plugins,
                            # 10 intentional defects; waits for health-check verdicts
npm run health              # PASS/FAIL for Admin API, proxy, seed state
```

Ports on the host: `8000` proxy, `8443` proxy TLS, `127.0.0.1:8001` Admin API,
`127.0.0.1:8002` Kong Manager. The Admin API is an unauthenticated control plane and is
bound to loopback only. Postgres is not published.

Other lifecycle commands: `npm run kong:down`, `npm run kong:reset` (destroys volumes),
`npm run kong:logs`, `npm run seed:reset` (deletes every seeded entity first; the seed is
idempotent either way).

What is seeded and why each defect exists: [docs/scenarios.md](docs/scenarios.md).

## Running the MCP server

```
npm run dev                 # tsx, stdio transport
# or
npm run build && npm start  # node dist/server.js
```

Register with an MCP client by pointing it at the built server. For Claude Desktop, in
`claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "kong-diagnostics": {
      "command": "node",
      "args": ["/absolute/path/to/kong-ai-gateway-mcp/dist/server.js"],
      "env": { "KONG_ADMIN_URL": "http://localhost:8001" }
    }
  }
}
```

The server needs no LLM key. It probes Kong at startup and logs a warning (not an error)
if the gateway is not yet up; tools return structured `UNREACHABLE` errors until it is.

Point clients at `node dist/server.js` (or `npx tsx src/server.ts`) directly, not at
`npm run ...`: npm prints a script banner to stdout, which is the protocol channel.
`npm run dev` is for driving the server by hand; it deliberately does not use watch mode
(see MCP-STDOUT-001 in the failure modes).

## Running the agent

Requires `GEMINI_API_KEY` in `.env`.

```
npm run agent -- "Why is authentication failing on /payments?"
npm run agent -- --trace "Clients hitting /orders get 429 immediately. What's going on?"
npm run agent -- --json "Is /shipping protected?"     # full run record as JSON
```

Exit code is 0 for a complete diagnosis and 1 when the run was cut off by a limit or a
provider error -- in which case the answer says so explicitly.

## Running evaluation

```
npm run eval                        # all 28 cases -> evals/results/run-NNN.{json,md}
npm run eval -- --case auth-001     # one case (repeatable flag)
npm run eval -- --difficulty hard
npm run eval -- --scenario UPSTREAM-UNHEALTHY-001
npm run eval -- --dry-run           # validate the dataset, run nothing
npm run eval -- --resume run-003    # finish a run the provider's daily quota cut short
```

## Configuration

All variables are validated at startup by [`src/config/env.ts`](src/config/env.ts).
Blank values fall through to defaults.

| Variable                     | Default                 | Purpose                                                                                                   |
| ---------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------- |
| `KONG_ADMIN_URL`             | `http://localhost:8001` | Admin API base URL                                                                                        |
| `KONG_PROXY_URL`             | `http://localhost:8000` | used only by `npm run health`                                                                             |
| `KONG_TIMEOUT_MS`            | `5000`                  | per-request timeout to Kong                                                                               |
| `LLM_PROVIDER`               | `gemini`                | provider for the agent and evaluation                                                                     |
| `GEMINI_API_KEY`             | _(unset)_               | required for `agent` and `eval` only                                                                      |
| `GEMINI_MODEL`               | `gemini-3.5-flash-lite` | any model the key can access; pricing must be declared in `src/telemetry/cost.ts` for cost to be reported |
| `LOG_LEVEL`                  | `info`                  | pino level; audit events always emit                                                                      |
| `LOG_FILE`                   | _(unset)_               | additional JSON log destination; stderr always                                                            |
| `ENABLE_WRITE_TOOLS`         | `false`                 | must stay false: no write tool is implemented and the server refuses to start otherwise                   |
| `MAX_TOOL_RESULT_ITEMS`      | `50`                    | cap on any list in a tool result                                                                          |
| `MAX_CONTEXT_CHARS`          | `12000`                 | cap on a single serialized tool result                                                                    |
| `MAX_AGENT_STEPS`            | `10`                    | LLM calls per run                                                                                         |
| `MAX_REPEATED_TOOL_CALLS`    | `2`                     | identical (tool, arguments) calls per run                                                                 |
| `MAX_TOTAL_TOOL_RESULT_SIZE` | `50000`                 | cumulative tool-result characters per run                                                                 |

`.env` is gitignored. Nothing in the project prints a key; `redactConfig` reduces it to
`[set]` / `[unset]`.

## Evaluation

**Methodology.** One case per user question; 28 cases across 11 scenarios and three
difficulties. Scoring is deterministic -- no LLM judge. A case passes when the
deterministic engine produced every expected finding code during the run, the answer
contains every expected fact, contains no named wrong conclusion, and the run finished
within its limits. Cases the provider's daily quota prevents from running are marked
SKIPPED and excluded from every rate rather than counted as failures.

**Results of the current run** (`run-003`, dataset 1.2.0, `gemini-3.5-flash-lite`,
Kong 3.9.3, 2026-09-12):

|                                          |                                                                                                                 |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Passed                                   | **28 / 28** (easy 6/6, medium 14/14, hard 8/8)                                                                  |
| Finding recall / fact match / completion | 100% / 100% / 100%                                                                                              |
| Steps per case                           | mean 3.5 (min 3, max 9)                                                                                         |
| Tool calls per case                      | mean 2.57; repeated calls: 0                                                                                    |
| Duration per case                        | median 5.4 s; mean 20.5 s (nine cases include 30-60 s of free-tier rate-limit backoff); retry-free median 4.5 s |
| Tokens                                   | 425,710 total, 15,204 per case                                                                                  |
| Estimated cost                           | $0.1496 total, $0.0053 per case                                                                                 |

Two earlier runs are kept: run-001 scored 22/28 and run-002 26/28, and in both every
miss was either a provider rate-limit loss or a correct answer the substring scorer
marked wrong. No case in any run failed because the agent gave a wrong diagnosis. How the
scorer was fixed, and what these numbers do and do not show, is in
[docs/evaluation.md](docs/evaluation.md). Per-case results and every verbatim answer:
[evals/results/run-003.md](evals/results/run-003.md).

## Observability

Structured JSON to stderr (and `LOG_FILE`), never stdout. Every tool call records
`requestId`, `traceId`, `taskId`, tool, duration, status, error category, result size and
whether it was a repeat; every LLM call records provider, model, duration,
provider-reported tokens, finish reason and estimated cost. A separate audit log records
each invocation with sanitized arguments at `warn`. Metrics (call counts, success rate,
mean / p50 / p95 latency, repeat rate, tokens, cost) are computed only from recorded
calls. Details: [docs/observability.md](docs/observability.md).

## Failure Modes

Twelve failures were observed while building this and are documented with symptom, root
cause, impact, mitigation and regression test -- including a halt path that would have
returned an empty answer, a cost calculation off by 10x, a per-day quota mistaken for a
rate limit, a Gemini 3 protocol requirement that broke every case after its first tool
call, and a scorer that marked correct answers wrong. Twenty-three more were deliberately
provoked and verified. [docs/failure-modes.md](docs/failure-modes.md).

## Security

[docs/security.md](docs/security.md) -- threat model, the five boundaries at which
secrets are stopped (each with its test), execution limits, the read-only guarantee and
how it is structural, what prompt injection is and is not mitigated, and what is not
handled.

## Architecture Decisions

[docs/decisions.md](docs/decisions.md) -- fourteen decisions in context / decision /
alternatives / trade-offs / consequences form, each tied to something in this repository.

## Context Management

[docs/context-management.md](docs/context-management.md) -- the mechanisms and the
measurements behind them.

## Testing

```
npm test                    # 140 unit tests; no Docker, no network
npm run test:integration    # 27 tests against live seeded Kong, incl. MCP over real stdio
npm run test:e2e            # 3 tests: question -> agent -> tool -> Kong -> diagnosis (needs a key)
npm run test:all
```

Integration and e2e suites skip themselves with a message when their dependency is
absent, so `npm test` passes on a fresh clone.

## Limitations

- **Read-only.** The agent can find a problem but not fix it. The write flow
  (confirmation token, preview diff, audit, rollback) is designed and documented, not
  built.
- **Rules must be written.** The engine detects the defect classes it has rules for
  (eleven). A novel class is invisible until someone adds a rule; the model cannot
  discover it from a summary that omitted the relevant field.
- **`find_route_by_path` matches whole paths.** It does not simulate Kong's prefix or
  regex routing.
- **One provider.** Gemini only. Anthropic and OpenAI are one file each away, but that
  file does not exist yet.
- **Free-tier constraints shaped the harness.** Per-minute rate limits add 30-60 s to some
  cases; some models have a 20-request daily quota. The harness handles both honestly but
  a paid tier would give cleaner latency numbers.
- **Scoring is substring-based.** It is a regression net for named facts and named wrong
  conclusions, not a judge of prose quality.
- **Prompt injection through gateway data is not detected.** It is defanged (no write
  path, typed arguments, facts from rules) but not detected or filtered.
- **Kong 3.9.x only** has been tested.
- **Sample size.** 28 cases, one model, one full run per dataset version. Enough to
  demonstrate and to catch regressions; not enough to state a stable percentage.

## Roadmap

Possible, not promised:

- Confirmation-gated write tools (enable a plugin, fix a route path) with preview and
  audit, behind `ENABLE_WRITE_TOOLS`.
- Anthropic and OpenAI providers.
- Rules for more defect classes: certificate/SNI mismatches, plugin ordering conflicts,
  consumer-scoped plugin overrides.
- A second full evaluation run per model to establish variance, and evaluation across
  models.
- OpenTelemetry export for tool and LLM spans.
- Human feedback on answers, feeding the case set.
- Other gateways behind the same tool contract.

## License

MIT.