MCP Control Plane

A hardened gateway that sits between AI agents (LLM clients) and real infrastructure APIs (Kubernetes, Terraform, Jenkins, Prometheus, ticketing). Every tool call is authenticated, rate-limited, policy-checked via OPA, and audited before it reaches an executor. The model is treated as an untrusted caller — the policy engine, not the prompt, is the actual security boundary.

Full target architecture, threat model, and API design live in docs/; this README covers what's implemented today and how to run it.

Status

Actively under development, tracked in docs/roadmap.md. Currently implemented:

• MCP JSON-RPC gateway (tools/list, tools/call) over HTTP, plus an SSE endpoint that also carries approval-decision pushes

• API key authentication, resolving each key to an agent identity (id, role, allowed_tools)

• Redis-backed sliding-window rate limiting, per agent

• Tool call interception, normalization, and an OPA policy check (allow / deny / require_approval)

• 7 read-only tool executors calling real downstream APIs (Kubernetes, Terraform Cloud, Jenkins, Prometheus, PagerDuty/Jira), plus the 3 write tools (restart_deployment, scale_deployment, trigger_jenkins_job) — docs/roadmap.md Phases 3-4.

• 2 higher-risk stretch tools, exec_into_pod and apply_k8s_manifest (docs/roadmap.md Phase 10): same Python kubernetes client, sre1-only, hard-blocked against kube-system, and gated behind approval in every environment (not just prod, unlike the write tools above).

• Human-in-the-loop approval gate for require_approval policy decisions: pending approvals are persisted to Redis with a 15-minute TTL, a Slack Incoming Webhook notifies with approve/deny buttons, and POST /admin/approvals/{id}/decide (HMAC-verified, replay-protected) resumes the tool call on approval or logs approval_denied on denial — see docs/roadmap.md Phase 4.

• PostgreSQL-backed, SHA-256 hash-chained audit log: every tool call (allowed, denied, errored, pending/decided approval) gets one append-only row via app/audit.py, schema-migrated with Alembic. GET /admin/audit queries it with filters/pagination and an integrity_check that recomputes the hash chain; GET /admin/audit/export streams the same data as NDJSON. A daily script/workflow exports the prior day's rows to S3 — see docs/roadmap.md Phase 5.

• OpenTelemetry traces + metrics across the whole request path (authn → policy_eval → executor → audit_write spans, tool_calls_total/policy_denials_total/ approval_requests_total/rate_limit_hits_total/tool_call_duration_ms and friends), an OTel collector + Tempo + Prometheus + Grafana stack in docker-compose.yaml, 6 provisioned Grafana dashboards and 4 alert rules under observability/ — see docs/roadmap.md Phase 6.

• Terraform for the full AWS deployment target (VPC, EKS + managed node group, ECR, RDS, ElastiCache Redis, ALB/ACM, IRSA IAM roles, Secrets Manager, an Object-Locked S3 bucket for the audit export) plus a Helm chart (helm/mcp-control-plane/) installed by Terraform itself, under terraform/ — see docs/roadmap.md Phase 7, including its "what's actually been checked" note: this was written with no AWS account available, so it's fmt/reference/init-clean and the Helm chart is lint/template-verified, but has never been plan/apply/destroy-run against real AWS.

Not yet built: admin JWT auth in front of /admin/approvals/* and /admin/audit* (both currently have no auth of their own beyond the approval-decide endpoint's Slack HMAC check), automated retention enforcement (the 90-day Postgres deletion and S3-to-Glacier lifecycle rule are documented but not scheduled anywhere), and the EKS DaemonSet variant of the OTel collector (Phase 6 only ships the Docker Compose one; terraform/helm.tf runs the collector as a regular in-cluster Deployment, not a DaemonSet) — see the roadmap for sequencing.

Related MCP server: Pincer-MCP

Architecture

Agent → POST /mcp → [log] → [authn] → [rate limit] → interceptor → OPA → executor → audit log

• app/main.py — FastAPI app, route handlers, middleware wiring

• app/middleware/auth.py — API key → agent identity

• app/middleware/rate_limit.py — Redis sliding-window limiter

• app/interceptor.py — validates/normalizes tools/call params into an OPA input document

• app/authz/opa.py — calls the OPA sidecar, maps its response to allow/deny/require_approval

• app/tools/tools_spec.py — tool executors, 7 read-only + 5 write (all call real downstream APIs)

• app/tools/k8s_client.py — cached K8s API clients (CoreV1Api/AppsV1Api + a generic DynamicClient for apply_k8s_manifest) + ApiException/timeout -> ExecutorError mapping

• app/tools/errors.py, app/tools/config.py — shared ExecutorError type and the 10s/30s executor timeout

• app/approvals.py — Redis-backed pending-approval store (create / decide, 15-minute TTL)

• app/slack.py — Slack Incoming Webhook notification + X-Slack-Signature HMAC verification

• app/sse_hub.py — in-process pub/sub pushing an approval decision to the agent's open /mcp/sse connection

• app/db.py — shared asyncpg (Postgres) connection pool

• app/audit.py — hash-chained audit log: async writer, filtered query, integrity check, NDJSON export

• migrations/ — Alembic migrations (raw SQL, no ORM) for audit_log and approvals

• scripts/export_audit_to_s3.py — daily S3 export of the prior day's audit rows

• policies/ — Rego policy (authz.rego) and role/tool data (data.json) loaded by OPA

See docs/architecture.md for the full target-state design (EKS, Terraform, Postgres audit log, approval gate, observability stack).

Tech stack

Python 3.13, FastAPI, Redis, OPA/Rego, Docker Compose. See pyproject.toml for exact dependency versions.

Getting started

Prerequisites

• Python 3.13+ and uv

• Docker (for Redis/Postgres/OPA via Compose), or your own local instances

Install

uv sync --extra export

(--extra export pulls in boto3 — tests/test_export_script.py imports scripts/export_audit_to_s3.py unconditionally, so a bare uv sync leaves the full uv run pytest run unable to even collect.)

Run

Everything (gateway + Redis + Postgres + OPA + the OTel collector/Tempo/Prometheus/Grafana observability stack) via Docker Compose:

docker compose up

Compose doesn't run the Alembic migration for you - apply it once, as a one-off container on the same network (postgres's port isn't published to the host by default, so this is simpler than reaching it from outside Compose):

docker compose run --rm gateway alembic upgrade head

Grafana is at http://localhost:3000 (anonymous admin access, no login screen - local dev only). The 6 dashboards and 4 alert rules under observability/grafana/ are provisioned automatically. The PagerDuty/Slack contact points (observability/grafana/provisioning/alerting/contact-points.yaml) ship with placeholder values and will provision cleanly either way - replace them with a real integration key/webhook URL before an alert needs to actually page anyone.

Dashboards start empty - they only show data once something sends real traffic through the gateway container (the one docker compose starts; only it has OTEL_EXPORTER_OTLP_ENDPOINT set - uv run pytest and a locally-run uvicorn deliberately don't export telemetry, see app/otel.py). With the stack up and migrated, fire a few requests at it (each maps to a different dashboard/panel) and wait ~60-90s for the metrics export + Prometheus scrape interval:

# Tool Call Volume + Latency - any tool call, success or failure, records these
curl -X POST http://127.0.0.1:8000/mcp -H "Content-Type: application/json" -H "x-api-key: test_key" \
  -d '{"jsonrpc":"2.0","id":"1","method":"tools/call","params":{"name":"get_pod_logs","arguments":{"namespace":"payments","pod_name":"checkout-api-xyz"}}}'

# Denial Rate - has to actually reach OPA and be denied there; a tool the agent isn't
# scoped to at all (app/middleware/auth.py's allowed_tools) gets rejected before OPA/audit
# ever see it and won't show up here - this hits OPA's kube-system hard-block instead
curl -X POST http://127.0.0.1:8000/mcp -H "Content-Type: application/json" -H "x-api-key: test_key" \
  -d '{"jsonrpc":"2.0","id":"2","method":"tools/call","params":{"name":"restart_deployment","arguments":{"namespace":"kube-system","deployment_name":"coredns","reason":"testing denial rate"}}}'

# Approval Queue - require_approval (any write tool in a prod-* namespace)
curl -X POST http://127.0.0.1:8000/mcp -H "Content-Type: application/json" -H "x-api-key: test_key" \
  -d '{"jsonrpc":"2.0","id":"3","method":"tools/call","params":{"name":"restart_deployment","arguments":{"namespace":"prod-payments","deployment_name":"checkout-api","reason":"testing approval queue"}}}'

# Rate Limit - test_key's default limit is 60/min (DEFAULT_RATE_LIMIT_RPM), so fire more than that quickly
for i in $(seq 1 70); do
  curl -s -o /dev/null -X POST http://127.0.0.1:8000/mcp -H "Content-Type: application/json" -H "x-api-key: test_key" \
    -d '{"jsonrpc":"2.0","id":"'$i'","method":"tools/list","params":{}}'
done

Audit Chain Integrity is the one dashboard these won't touch - it only reports data when GET /admin/audit actually finds a broken hash chain, so staying at zero is the correct/healthy state, not something to artificially populate. On Windows, run the block above from PowerShell or Git Bash, not cmd.exe - cmd.exe doesn't treat '...' as a string delimiter the way these commands assume, so the JSON body arrives with literal quote characters in it and the gateway correctly rejects it with -32700 Parse error (same failure mode testing/testing.text 3.8 tests on purpose, just not on purpose here).

Or the gateway alone against local dependencies (Redis, Postgres, OPA each running separately). Apply the audit-log schema once before the first run:

uv run alembic upgrade head
uv run uvicorn app.main:app --host 127.0.0.1 --port 8000 --reload

The app reads REDIS_URL (default redis://localhost:6379), DATABASE_URL (default postgresql://postgres:postgres@localhost:5432/mcp_control_plane), and OPA_URL (default http://localhost:8181/v1/data/authz) from the environment; Compose sets all three to point at the sibling containers. alembic upgrade head reads the same DATABASE_URL.

Tool executor configuration

Each read-only executor talks to a real downstream API and reads its own credentials/URL from the environment (all optional locally — a tool without its config set returns an upstream_error rather than crashing):

Local Kubernetes tools: point KUBECONFIG at a local cluster (e.g. kind create cluster or minikube start, then kubectl config use-context kind-kind / minikube) before calling get_pod_logs, list_pods, or get_deployment_status — the gateway uses whatever context is currently active, same as kubectl.

Approval gate configuration

The pending-approval record itself lives in the same Redis as the rate limiter (REDIS_URL), with a 15-minute TTL — no separate store to configure.

Audit log S3 export configuration

scripts/export_audit_to_s3.py (run daily by .github/workflows/audit-export.yml) needs:

The GitHub Actions workflow reads these from AUDIT_LOG_S3_BUCKET/AUDIT_LOG_DATABASE_URL/etc. repo secrets and no-ops if they're not configured — there's no deployed Postgres/S3 for this project yet (Phase 7, Infrastructure as Code, hasn't started).

Try it

All /mcp routes require an x-api-key header. The seeded dev key test_key resolves to agent01 (role sre1) with a handful of allowed tools — see API_KEYS in app/middleware/auth.py.

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" -H "x-api-key: test_key" \
  -d '{"jsonrpc":"2.0","id":"1","method":"tools/list","params":{}}'

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" -H "x-api-key: test_key" \
  -d '{"jsonrpc":"2.0","id":"2","method":"tools/call","params":{"name":"get_pod_logs","arguments":{"namespace":"payments","pod_name":"checkout-api-xyz"}}}'

curl -s http://127.0.0.1:8000/health/live

More request/response examples (error cases, SSE) are in testing/testing.text.

Testing

uv run pytest

Covers the auth middleware, rate limiter, OPA integration, and end-to-end /mcp request handling (see tests/). By default record_tool_call/create_db_pool are stubbed (no real Postgres needed), the same way evaluate_policy is stubbed to allow everything (no real OPA needed) — see tests/conftest.py. Two subsets need Docker and are skipped automatically if it isn't available:

• tests/test_opa_docker_integration.py — real openpolicyagent/opa image against policies/

• tests/test_audit_docker_integration.py — real postgres:16-alpine, with the actual alembic upgrade head migration applied, exercising the hash chain (including a tamper/detect test) and the query/export endpoints for real

Rego policy tests run separately, via opa test:

docker run --rm -v "$(pwd)/policies:/policies" openpolicyagent/opa:latest test /policies -v

On Windows, run that from PowerShell, not Git Bash — Git Bash's MSYS layer auto-translates the leading /policies in -v "$(pwd)/policies:/policies" into a bogus Windows path (stat /Program Files/Git/policies: no such file or directory), which has nothing to do with the policies themselves:

docker run --rm -v "${PWD}\policies:/policies" openpolicyagent/opa:latest test /policies -v

opa test runs in CI on every PR (see .github/workflows/ci.yml); the Python test suite above is not currently wired into CI.

Documentation