azure-compliance-mcp

Honest Azure compliance for LLM agents — ask infra-health questions in plain English and get answers that distinguish a real fail from a not_evaluable, and never fake a pass.

An MCP server, built with FastMCP 3.x, that exposes read-only Azure resource-compliance data so an agent can answer questions like "which storage accounts allow TLS 1.0?" or "which VMs have no disk encryption?" — each finding tagged with where the verdict came from and whether it could be evaluated at all.

Why honest compliance?

Most tooling collapses "compliant", "non-compliant", and "couldn't tell" into a single pass/fail bit — so a control it can't actually measure silently shows green. This server refuses to guess:

• fail — positive evidence a protection is absent (e.g. encryptionAtHost = false). Actionable; shows up in the default view.

• not_evaluable — the signal genuinely isn't in the data (e.g. no guest-config policy assigned). Surfaced explicitly, never as a pass.

• source — every finding records the signal that decided its verdict: arg | azure_policy | graph | defender.

Related MCP server: Purple AI MCP Server

Quickstart

Requires uv and Python 3.12+. Runs out of the box — mock mode needs zero Azure setup.

uv sync                          # install dependencies
uv run python scripts/demo.py    # fastest look — runs the tools against mock data
uv run server.py                 # mock data, stdio transport — works immediately

Point it at your own tenant when you're ready:

uv run server.py --mode live     # real Azure Resource Graph (e.g. after `az login`)
uv run server.py --transport http  # remote: Streamable HTTP
uv run fastmcp dev inspector server.py  # explore the tools interactively

Live mode authenticates with DefaultAzureCredential against your own tenant and is strictly read-only.

Architecture

Tools depend only on a Provider protocol, never on a concrete data source — so the same tool code runs against synthetic mock data or live Azure Resource Graph (ARG), selected at startup with --mode.

flowchart LR
    Agent["LLM agent / MCP client"] -->|stdio · HTTP| Server["FastMCP server\n(5 read-only tools)"]
    Server --> Protocol["Provider protocol\n(read-only)"]
    Protocol -->|--mode mock| Mock["MockProvider\nseeded ARG-shaped data"]
    Protocol -->|--mode live| Live["LiveProvider\nDefaultAzureCredential"]
    Live -->|injection-safe KQL| ARG[("Azure Resource Graph\nresources · policyresources ·\npatchassessmentresources · authorizationresources")]
    Live -->|getByIds| Graph[("Microsoft Graph\nprincipal existence")]

Controls (check_compliance evaluates five):

Tools (5 of a 5–6 budget):

Transports: stdio (default, local) and Streamable HTTP (remote, intended behind OAuth 2.1). In stdio mode stdout is reserved for the protocol — logging goes to stderr only.

See SPEC.md for the full tool contracts.

Design decisions & tradeoffs

• Mock provider is the default. The repo runs with zero Azure account, credentials, or network — so reviewers, CI, and new contributors get meaningful output immediately. The mock dataset is shaped exactly like ARG rows, so one mapping serves both modes.

• Read-only by construction. No tool can create, update, or delete a resource. The live path only ever issues ARG queries; there is no mutation surface to misuse.

• Control-based, not Azure-Policy-based. Compliance is evaluated from resource configuration (the ARG row), so it works even when no Azure Policy is assigned — except where the signal genuinely lives elsewhere (guest-config), which routes to policyresources and is honest about it.

• The not_evaluable + source honesty model. A third status plus a provenance field means the agent can tell "this is broken" from "I can't see this from here" — the core reason to trust the output.

• KQL pushdown for scale. Live filters (query_resources) are pushed into the ARG query (where + take) rather than fetched-then-filtered, so large tenants stay cheap. An opt-in contract test asserts the pushdown matches the reference Python filter, so both modes stay provably consistent.

• Injection-safe escaping. ARG has no bind parameters, so every user-supplied value is encoded as an escaped KQL string literal — never concatenated raw. (ARG is read-only regardless, but the discipline is enforced and unit-tested.)

Status & roadmap

v0.3.0 — check_compliance, query_resources, get_patch_status, and find_orphaned_rbac shipped (mock + live), honest not_evaluable/source model across all of them, KQL pushdown, CI on every PR.

Next:

• summarize_health — rolled-up infra-health summary

• evals — task-level evaluation of agent answers over the mock dataset

Development

uv run pytest               # tests (live tests are opt-in: RUN_LIVE_TESTS=1)
uv run ruff check .         # lint
uv run ruff format --check . # format

Evals (opt-in)

Tests prove the tools behave; the eval suite (SPEC §5) checks the property the server exists for: given a natural-language question, does an agent call the right tool, with the right arguments, and answer correctly — including saying "that can't be determined" when the honest answer is not_evaluable.

# needs ANTHROPIC_API_KEY (env or the gitignored .env); calls the API — costs money
uv run --group evals python evals/run.py                     # full suite (~20 cases)
uv run --group evals python evals/run.py --cases rbac        # filter by case id
uv run --group evals python evals/run.py --agent-model <id>  # evaluate another agent model

# alternate provider for cost-free smoke runs (needs OPENROUTER_API_KEY);
# a non-Anthropic judge is OFF-PIN vs SPEC §5.4 — the report flags it
uv run --group evals python evals/run.py --provider openrouter \
    --agent-model <openrouter-id> --judge-model <openrouter-id>

Runs against mock mode only (deterministic, no Azure credentials). Results land in evals/results/<timestamp>-<model>.json plus a rendered summary; they're gitignored except deliberately committed baseline-* files. Structured assertions are exact-match; open-ended prose is scored by a pinned LLM judge whose ~85–92% human-agreement caveat is printed in every report — judge scores are signal, not ground truth. Deliberately not in CI (SPEC §5.6).

Security

• All Azure tools are read-only — nothing in this server modifies Azure resources.

• Secrets, tenant IDs, and .env are gitignored and must never be committed; live mode uses only your local DefaultAzureCredential.

• In stdio mode, logging goes to stderr only (stdout is reserved for the protocol).

License

MIT