Vault MCP Bridge

Vault MCP Bridge

Overview

• FastAPI MCP-compatible server that manages agent-scoped secrets and crypto via HashiCorp Vault.

• Auth: API Key, JWT (HS256 and RS256/JWKS), and mTLS via reverse-proxy headers.

• Per-agent namespacing in KV v2; Transit support (encrypt/decrypt/sign/verify/rewrap/random).

• Optional per-request child token issuance bound to per-agent policies; simple per-agent rate limiting.

• Prometheus metrics at /metrics and optional OpenTelemetry tracing.

Quickstart

• Python env:

  • python3 -m venv .venv && source .venv/bin/activate

  • python -m pip install -r requirements.txt

• Start local Vault (dev) for testing:

  • cd local-vault && docker compose up -d && cd ..

  • This provisions KV v2 at secret/, a Transit key, and example policies. See local-vault/README.md.

• Run server:

  • Easiest: python main.py (env: HOST=0.0.0.0 PORT=8089 RELOAD=true LOG_LEVEL=debug)

  • Or: python -m uvicorn main:app --reload

  • Or factory: python -m uvicorn vault_mcp.app:create_app --factory --reload

• Helper: scripts/run_dev_jwt.sh starts the app with Vault + JWT defaults (no manual env export).

• Sanity checks:

  • curl http://127.0.0.1:8089/healthz

  • bash scripts/smoke.sh (expects local Vault and default dev auth)

• Vault access (required before running the server):

  • Set credentials so the app can authenticate to Vault, e.g. for the bundled dev compose stack:

    export VAULT_ADDR=http://127.0.0.1:8200 export VAULT_TOKEN=root # replace with your own token or AppRole

  • Without these env vars (or the AppRole equivalents) /readyz will return 503 and observability/secret routes will be unauthorized.

  • To avoid re-exporting each run you can source config/dev-jwt.env (new sample file) before launching, or use scripts/run_dev_jwt.sh which applies the same defaults automatically.

• Optional admin UI:

  • python3 -m venv .ui-venv && source .ui-venv/bin/activate

  • pip install -r ui/requirements.txt

  • streamlit run ui/streamlit_app.py

  • Use the Manage Subjects & Keys page to manage config/users.json, rotate credentials, and sync profiles to the sidebar.

  • JWT helpers in the UI require JWT_HS256_SECRET to be set before launch (e.g., source config/dev-jwt.env or run scripts/run_dev_jwt.sh).

  • Use AWS KMS Operations to supply temporary AWS credentials (if needed) and call encrypt/decrypt, data-key, sign, and verify APIs.

JWT quickstart (HS256):

• Install helper deps: pip install 'python-jose[cryptography]'

• Generate a token with the bundled script (adjust subject/scopes as needed):

  • python scripts/gen_jwt.py \ --secret dev-secret \ --issuer mcp-auth \ --audience mcp-agents \ --sub agent_api \ --scopes read,write,delete,list \ --ttl 600

• Use the printed value as the Authorization: Bearer <token> header when calling the API or configuring the Streamlit console.

Features

• KV v2 secret CRUD with per-agent prefixes and safe pathing.

• Transit: encrypt/decrypt, sign/verify, rewrap, and random bytes (base64/hex).

• AWS KMS (optional): encrypt/decrypt, generate data keys, and sign/verify with native AWS keys.

• Database: dynamic credentials issuance and lease management.

• SSH: OTP credential and SSH certificate signing.

• Auth: API Key, JWT (HS256 or RS256 via JWKS), mTLS via headers.

• Child tokens per request (optional); per-agent in-memory rate limiting.

• MCP: JSON-RPC over HTTP at POST /mcp/rpc (with GET /mcp/sse keepalive channel) and stdio transport via scripts/mcp_stdio.py.

• Streamlit operations hub: Vault Operations covers Secrets, Transit, Database, SSH, and MCP tools, while a dedicated AWS KMS page lets you enter temporary AWS credentials and invoke encrypt/decrypt, data-key, and signing APIs.

• Streamlit agent admin: create multiple AI agent profiles, toggle LLM usage, assign credentials (linked user/API key/JWT), define tasks, capture a secrets_backend plan (Vault/KMS/Hybrid), and monitor progress/status.

• MCP lifecycle: initialize, tools/list, resources/list, prompts/list, tools/call, shutdown. Protocol version: 2025-06-18.

• Tools exposed (with required scopes):

  • KV: kv.read (read, supports version), kv.write (write), kv.list (list), kv.delete (delete), kv.undelete (write), kv.destroy (write)

  • Transit: transit.encrypt (write), transit.decrypt (read), transit.sign (write), transit.verify (read), transit.rewrap (write), transit.random (read)

  • DB: db.issue_creds (write), db.renew (write), db.revoke (write)

  • SSH: ssh.otp (write), ssh.sign (write)

  • KMS: kms.encrypt (write), kms.decrypt (read), kms.generate_data_key (write), kms.sign (write), kms.verify (read)

• Observability endpoints: /observability/summary (Vault/API status + in-flight requests) and /observability/logs/{requests|responses|server} (tail JSON logs, read scope required).

• Metrics at /metrics; optional OpenTelemetry via OTLP HTTP exporter.

Resources

• Scheme kv://{subject}/{path} with optional ?version=N.

• resources/list: advertises kv://{subject}/ (KV root) for the authenticated subject.

• resources/get:

  • kv://{subject}/foo/bar returns { data, version } (JSON) for that KV path.

  • kv://{subject}/foo/ (trailing slash) returns { keys: [...] } listing under that prefix.

  • Cross-subject access is forbidden.

Prompts

• prompts/list: returns prompt specs for kv_read and kv_write with input schemas.

• prompts/get:

  • kv_read: returns example messages and a suggested_tool call for kv.read.

  • kv_write: returns example messages and a suggested_tool call for kv.write.

Configuration (env)

• Vault:

  • VAULT_ADDR (default: http://localhost:8200)

  • VAULT_NAMESPACE (Enterprise only)

  • VAULT_TOKEN or VAULT_ROLE_ID + VAULT_SECRET_ID

  • KV_MOUNT (default: secret)

  • DEFAULT_PREFIX (default: mcp)

• Config file (optional, no .env needed):

  • Set APP_CONFIG_FILE to a JSON/TOML/YAML file path. Example defaults auto-detected from CWD: config.toml, config.json, config.yaml.

  • Environment variables always override file values.

  • Note: .env files are not auto-loaded anymore.

  • Precedence: runtime args (where applicable) → environment variables → APP_CONFIG_FILE/auto-detected config → built-in defaults.

• Auth enable flags:

  • AUTH_API_KEY_ENABLED (default: true)

  • AUTH_JWT_ENABLED (default: true)

  • AUTH_MTLS_ENABLED (default: false)

  • CLI helper: scripts/manage_user.py create <subject> writes metadata to config/users.json and prints policy/API key export commands.

  • UI helper: in Streamlit, check “Generate JWT token” to issue a token during user creation. Generation failures show inline errors and the user entry is not written.

• API Keys:

  • API_KEYS_JSON JSON map: { "<api-key>": "<agent-id>" }

• JWT:

  • Common: JWT_ISSUER (default: mcp-auth), JWT_AUDIENCE (default: mcp-agents)

  • HS256: JWT_HS256_SECRET

  • RS256/JWKS: JWT_JWKS_URL or JWT_JWKS_FILE, JWT_JWKS_CACHE_SECONDS (default: 300), JWT_REQUIRE_KID (default: false)

  • Validation toggles: JWT_VALIDATE_ISSUER (default: true), JWT_VALIDATE_AUDIENCE (default: true)

  • Helper: python scripts/gen_jwt.py --secret <JWT_HS256_SECRET> --sub <agent> for quick dev tokens (see Quickstart example above).

  • Metadata persisted per user: jwt_created_at, jwt_expires_at, jwt_ttl_seconds. The Streamlit Current users grid surfaces status, timestamps, TTL seconds, and offers CSV export.

• mTLS via proxy headers:

  • MTLS_IDENTITY_HEADER (default: x-ssl-client-s-dn)

  • MTLS_VERIFY_HEADER (default: x-ssl-client-verify)

  • MTLS_SUBJECT_CN_PREFIX (default: CN=)

• Child token issuance:

  • CHILD_TOKEN_ENABLED (default: false)

  • CHILD_TOKEN_TTL (default: 90s)

  • CHILD_TOKEN_POLICY_PREFIX (default: mcp-agent-)

• Rate limiting:

  • RATE_LIMIT_ENABLED (default: true)

  • RATE_LIMIT_REQUESTS (default: 60)

  • RATE_LIMIT_WINDOW_SECONDS (default: 60)

• Server behavior:

  • HOST, PORT, RELOAD, LOG_LEVEL for python main.py

  • Logs directory: LOG_DIR (default: ./logs)

  • EXPOSE_REST_ROUTES (default: true) — when false, disables REST feature routers (/secrets, /transit, /db, /ssh, /whoami) for MCP‑only deployments.

  • AWS KMS (optional):

    • AWS_KMS_ENABLED (default: false)

    • AWS_REGION (required when enabling KMS unless provided by IAM role/default profile)

    • AWS_KMS_DEFAULT_KEY_ID (optional convenience default for encrypt/data-key/sign requests)

    • AWS_KMS_ENDPOINT (optional; point to LocalStack or custom endpoint)

    • AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional; falls back to standard AWS credential resolution)

• Observability:

  • Prometheus at /metrics (always enabled)

  • OpenTelemetry: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME (optional)

Auth Modes

• API Key: send X-API-Key: <key>. Map keys to agents via API_KEYS_JSON.

• JWT: send Authorization: Bearer <token> with sub and optional scopes.

• mTLS: terminate TLS at proxy and pass DN via X-SSL-Client-S-DN; CN is used as subject.

Agent Path Namespace

• Secrets live under {KV_MOUNT}/data/{DEFAULT_PREFIX}/{subject}/... (KV v2). The server enforces safe relative paths within the agent prefix.

Child Token Issuance

• If CHILD_TOKEN_ENABLED=true, a child token is minted per request with policy {CHILD_TOKEN_POLICY_PREFIX}{subject} and TTL CHILD_TOKEN_TTL.

• Ensure the policy exists and the parent token can create child tokens.

Policy

• Generate HCL for an agent:

  • python scripts/gen_policy.py --agent alice --mount secret --prefix mcp > alice.hcl

  • Suggested policy name: mcp-agent-alice

• The policy grants CRUD/list on data/{prefix}/{agent}/*, list/read on metadata/{prefix}/{agent}/*, and versioned ops on delete/undelete/destroy.

• Apply with the Vault CLI: vault policy write mcp-agent-alice alice.hcl

Endpoints

• KV v2

  • PUT /secrets/{path} — write (scope: write)

  • GET /secrets/{path} — read (scope: read) [query version optional]

  • DELETE /secrets/{path} — delete latest version (scope: delete)

  • GET /secrets?prefix=... — list keys (scope: list)

  • POST /secrets/{path}:undelete — body { "versions": [1,2] } (scope: write)

  • POST /secrets/{path}:destroy — body { "versions": [1,2] } (scope: write)

• Transit

  • POST /transit/encrypt — { "key": "k", "plaintext": "<b64>" } (scope: write)

  • POST /transit/decrypt — { "key": "k", "ciphertext": "..." } (scope: read)

  • POST /transit/sign — { key, input, hash_algorithm?, signature_algorithm? } (scope: write)

  • POST /transit/verify — { key, input, signature, hash_algorithm? } (scope: read)

  • POST /transit/rewrap — { key, ciphertext } (scope: write)

  • GET /transit/random?bytes=32&format=base64|hex (scope: read)

• KMS endpoints are always exposed for development but return 503 unless AWS_KMS_ENABLED=true

  • POST /kms/encrypt — { plaintext: <b64>, key_id?, encryption_context?, grant_tokens? } (scope: write)

  • POST /kms/decrypt — { ciphertext: <b64>, encryption_context?, grant_tokens? } (scope: read)

  • POST /kms/data-key — { key_id?, key_spec? | number_of_bytes?, encryption_context?, grant_tokens? } (scope: write)

  • POST /kms/sign — { key_id?, message? | message_digest?, signing_algorithm, message_type?, grant_tokens? } (scope: write)

  • POST /kms/verify — { key_id?, signature: <b64>, message? | message_digest?, signing_algorithm, message_type?, grant_tokens? } (scope: read)

• Database

  • POST /db/creds/{role} — issue dynamic DB creds (scope: write)

  • POST /db/renew — { lease_id, increment? } (scope: write)

  • POST /db/revoke — { lease_id } (scope: write)

• SSH

  • POST /ssh/otp — { role, ip, username, port? } (scope: write)

  • POST /ssh/sign — { role, public_key, cert_type?, valid_principals?, ttl? } (scope: write)

• Health/Debug/Metrics

• GET /healthz, /livez, /readyz, /whoami, /echo-headers, /metrics

  • /readyz returns a JSON body detailing Vault (ok/detail) and, when enabled, AWS KMS readiness. Expect HTTP 503 with explanatory messages when Vault credentials are missing or KMS is misconfigured.

• MCP

  • Mounted at /mcp when fastapi-mcp is available

API Docs

• Swagger UI: http://127.0.0.1:8089/docs

• ReDoc: http://127.0.0.1:8089/redoc

• OpenAPI: http://127.0.0.1:8089/openapi.json

MCP Usage

• HTTP JSON-RPC: POST /mcp/rpc with JSON-RPC 2.0 messages; authenticate same as REST (API key / JWT / mTLS).

• SSE channel: GET /mcp/sse provides periodic keepalives for server→client messaging (placeholder; extend as needed).

• stdio: SUBJECT=agentA python scripts/mcp_stdio.py and write newline-delimited JSON-RPC messages to stdin.

• Initialize result includes protocolVersion: 2025-06-18, basic capabilities, and lists tools/resources/prompts.

• SSE events: server emits tool.completed events with {type, tool, subject, ts}; keepalives every 15s.

MCP Inspector

• An official, interactive UI for MCP servers (Swagger-like, but for MCP) that discovers tools/resources/prompts and lets you call them live.

• Connect via HTTP:

  • RPC URL: http://127.0.0.1:8089/mcp/rpc

  • SSE URL (optional): http://127.0.0.1:8089/mcp/sse

  • Auth headers: add X-API-Key: dev-api-key (or Authorization: Bearer <JWT>) in the Inspector’s connection settings.

  • If connecting from the hosted Inspector (HTTPS) to your local HTTP server, enable CORS:

    • export CORS_ALLOW_ORIGINS=https://inspector.modelcontextprotocol.io

    • Consider exposing your server via HTTPS (e.g., ngrok) to avoid mixed-content blocking.

• Or connect via stdio:

  • Command: SUBJECT=agent_api python scripts/mcp_stdio.py

  • Inspector will spawn the process and speak JSON-RPC over stdio.

• Once connected, Inspector should list the available tools: kv.read, kv.write, kv.list, kv.delete, kv.undelete, kv.destroy.

• Current state: Resources/Prompts are empty; SSE sends keepalives only.

Troubleshooting Inspector

• ModuleNotFoundError: vault_mcp when running stdio: ensure you run from repo root, or use SUBJECT=agent_api PYTHONPATH=$(pwd) python scripts/mcp_stdio.py. The script now auto-adds repo root to sys.path.

• CORS errors in the browser: set CORS_ALLOW_ORIGINS=https://inspector.modelcontextprotocol.io (comma separate multiple origins) and restart the server.

• Mixed content blocked: use an HTTPS tunnel to your local server (e.g., ngrok http 8089) and switch Inspector URLs to https.

Prometheus & OpenTelemetry

• Prometheus endpoint: GET /metrics (text). Quick check: curl -s http://127.0.0.1:8089/metrics | head.

• Metrics include http_requests_total and http_request_duration_seconds with labels method, route, status.

• Additional telemetry: http_requests_with_correlation_total counts requests that include or receive a correlation ID. Every HTTP response returns X-Correlation-Id; if OTEL tracing is enabled, X-Trace-Id is also emitted.

• OpenTelemetry tracing (optional): set OTEL_EXPORTER_OTLP_ENDPOINT (e.g., http://localhost:4318/v1/traces) and OTEL_SERVICE_NAME (default: vault-mcp).

• Structured logs: JSON files under ./logs/ (requests.log, responses.log, server.log). Tail with tail -f logs/requests.log.

Logging Details

• Format: newline-delimited JSON. Core fields: ts, lvl, msg, logger plus context in extra.

• Request logs (vault_mcp.request): include request_id, client, method, path, status, duration_ms.

• Response/event logs (vault_mcp.response): per-endpoint keys, e.g.,

  • kv_put|kv_get|kv_delete: subject, path, keys, version, request_id

  • kv_list: subject, prefix, count, request_id

  • transit_*: subject, key, size/validity hints, request_id

  • db_* and ssh_*: high-level descriptors (e.g., role, lease_id_suffix, ip, user), never secret values

• Request ID: responses include X-Request-Id; it is echoed in logs for correlation.

• Example request log line:

  • { "ts": "2024-01-01T10:00:00", "lvl": "info", "msg": "request", "logger": "vault_mcp.request", "request_id": "...", "client": "127.0.0.1", "method": "GET", "path": "/secrets", "status": 200, "duration_ms": 12 }

Examples (curl)

• Write then read a secret (API key dev-key for agent agent_api):

  • curl -X PUT -H 'X-API-Key: dev-key' -H 'Content-Type: application/json' \ -d '{"data": {"foo":"bar"}}' http://127.0.0.1:8089/secrets/configs/demo

  • curl -H 'X-API-Key: dev-key' http://127.0.0.1:8089/secrets/configs/demo

• Random bytes from Transit (hex):

  • curl -H 'X-API-Key: dev-key' 'http://127.0.0.1:8089/transit/random?bytes=16&format=hex'

• RS256/JWKS quick test:

  • See local-vault/jwks/README.md for generating keys, running JWKS, and testing.

Local Dev Helpers

• Start server with sensible dev env: bash scripts/run_dev.sh

• Enable all auth regardless of current env: bash scripts/run_all_auth.sh

• Smoke test (server + local Vault): bash scripts/smoke.sh

• Auth tests by agent:

  • API key (agent_api): bash scripts/test_agent_api.sh

  • JWT HS256 (agent_jwt): bash scripts/test_agent_jwt.sh

  • JWT RS256/JWKS (agent_jwt): bash scripts/test_agent_jwt_rs256.sh

  • mTLS headers (agent_mtls): bash scripts/test_agent_mtls.sh

End-to-End and Example Agents

• One-shot E2E (server + MCP HTTP + stdio + optional REST):

  • LOG_LEVEL=DEBUG bash scripts/e2e_local.sh

• Example MCP agents (HTTP JSON-RPC):

  • API key: bash scripts/run_example_agent.sh

  • JWT: bash scripts/run_example_agent.sh --jwt 'YOUR_JWT'

  • mTLS headers: bash scripts/run_example_agent.sh --mtls

  • No‑LLM direct client (no model provider): bash scripts/run_example_agent.sh --no-llm

Examples

• LangChain agent that wraps these endpoints as tools:

  • See examples/langchain_agent/README.md and examples/langchain_agent/agent.py

Testing

• Run tests: pytest

• Pytest overview:

  • tests/test_health.py: Verifies basic liveness endpoints — GET /healthz and GET /livez return ok: true.

  • tests/test_auth_and_kv.py: Exercises API‑key auth and KV v2 CRUD.

    • Uses a mocked hvac KV client to avoid real Vault.

    • Flow: PUT /secrets/configs/demo writes data, GET reads it back, DELETE removes it, subsequent GET returns 404.

    • Also checks GET /whoami returns the expected subject for X-API-Key.

  • tests/test_transit_random.py: Tests Transit random byte generation endpoint with deterministic mock.

    • Monkeypatches client_for_principal to return a stub where generate_random_bytes is predictable.

    • Validates both format=hex and default base64 responses for GET /transit/random.

  • tests/test_health_ready.py: Covers /readyz for authenticated, unauthenticated, Vault error, and generic error cases.

  • tests/test_kv_extras.py: Covers GET /secrets?prefix=... list and version ops (:undelete, :destroy).

  • tests/test_transit_endpoints.py: Covers /transit/encrypt|decrypt|sign|verify|rewrap with a transit stub.

  • tests/test_db_and_ssh_routes.py: Covers /db/creds|renew|revoke and /ssh/otp|sign with stubs.

  • tests/test_auth_modes.py: HS256 JWT /whoami (valid and bad aud), mTLS header success/fail.

  • tests/test_auth_jwt_rs256_local.py: Local RS256: generates RSA + JWKS and validates /whoami via monkeypatched JWKS.

  • tests/test_rate_limit_and_metrics.py: Verifies /metrics and rate limiting on /transit/random (429 on third call).

  • tests/test_security_path_and_scopes.py: Path sanitization and 403 when scopes are insufficient.

  • tests/test_app_exception_handlers.py: Maps Vault Forbidden -> 403 and VaultError -> 502 JSON responses.

  • (If you add MCP client tests) exercise POST /mcp/rpc for initialize, tools/list, and tools/call with a JWT or API key.

Run subsets

• Keyword filter: pytest -k transit

• Coverage detail: pytest -q --cov=vault_mcp --cov-report=term-missing

Security Notes

• Use TLS end-to-end; for mTLS, terminate at a trusted proxy and pass identity headers.

• Avoid logging secret values; the app uses structured logging with response metadata only.

• Prefer JWT or mTLS in production; reserve API keys for development.

• Enable Vault audit devices and keep token TTLs minimal.

Troubleshooting

• Import errors (e.g., fastapi not found): ensure you use the same Python interpreter that installed deps.

  • python -m uvicorn main:app --reload

• Uvicorn targets: use <module>:<attribute> — e.g., main:app.

• Change port/host: python -m uvicorn main:app --reload --port 8090 --host 0.0.0.0

• Increase logs: add --log-level debug --access-log