Compliance Evidence Automation Agent

That badge gates five things, not just "the tests pass": gap detection is exact against the planted deficiencies and the existence-only arm still finds none of them; every tamper class is caught by its expected detector with no no-op tampers; every credential revocation produces zero silent gaps while the existence-only arm produces some; the independent auditor accepts every audit-ready control and still separates artifact quality from control effectiveness; and removing any one of the twenty guards makes a specific test fail.

An agent that collects SOC 2 evidence for 15 controls out of four company systems over scoped MCP tools, stores every artifact with a hash-chained provenance record, and then refuses to call a control done when the evidence is stale, partial, misaimed, or unattested. It finds all six planted deficiencies with the right reason for each. The same evidence through an existence-only checker reports 15 of 15 green — including through a credential revocation that blinded it completely.

Why this project exists

Every quarter, somebody assembles hundreds of evidence artifacts for an audit. It is repetitive, multi-system, correctness-critical work — exactly what agents are for. The risk is also exactly what agents are bad at: a collection pipeline that fails quietly produces a dashboard that is green and wrong, and nobody finds out until an auditor does.

So this is built around the three ways that happens, and each one is measured:

1. Collecting evidence today does not make it current. An artifact carries effective_at — the day the review was signed, the scan completed, the restore ran — separately from collected_at. The corpus contains a real, signed, population-complete access review attestation from May 2025 that the collector happily fetches today. Freshness is measured from the first field. A tracker keyed on the second reports it green forever.

2. Most compliance evidence is an absence claim, and absence inverts badly. "No user lacks MFA" and "I can no longer see users" are the same HTTP 200. Directory and ticketing APIs apply permissions as a row filter, so a revoked scope returns fewer rows rather than a 403. Every endpoint that can do that has a count endpoint on a different scope, and no collector may emit a clean artifact from zero observations.

3. Complete evidence of a failing control is not a pass. A vulnerability scan that ran on time and honestly reported four criticals past SLA is perfect evidence and a broken control. Tools that track collection report it green; evidence completeness and control effectiveness are kept apart here and reported separately.

Related MCP server: SOC2 Compliance AI MCP

Status

Built and measured. Every number below comes from a committed script running against the deterministic system stand-ins, reproducible with the commands in Reproduce.

The headline: same evidence, two checkers

Fifteen controls, one collection run, two ways of asking "is this done".

control   criteria  system     effective    age    coverage  this checker            existence-only
CC6.1-a   CC6.1     idp        2026-06-30     1d   62/62     READY                   READY
CC6.1-b   CC6.1     idp        2026-06-30     1d     7/7     READY                   READY
CC6.2     CC6.2     idp        2026-06-24     7d     9/9     READY                   READY
CC6.3     CC6.3     idp        2025-05-12   415d     1/1     GAP stale               READY   <-
CC6.6     CC6.6     cloud      2026-06-30     1d   14/14     READY                   READY
CC6.7     CC6.7     cloud      2026-06-30     1d   40/52     GAP population_incomplete READY <-
CC6.8     CC6.8     idp        2026-06-30     1d   62/62     READY                   READY
CC7.1     CC7.1     ci         2026-06-27     4d     1/1     GAP control_ineffective READY   <-
CC7.2     CC7.2     cloud      2026-06-30     1d     1/1     READY                   READY
CC7.3     CC7.3     ticketing  2026-06-27     4d   11/11     READY                   READY
CC7.4     CC7.4     ticketing  2026-05-19    43d     1/1     GAP unsigned            READY   <-
CC8.1-a   CC8.1     ci         2026-06-30     1d   47/47     GAP control_ineffective READY   <-
CC8.1-b   CC8.1     ci         2026-06-30     1d     1/1     READY                   READY
A1.2-a    A1.2      cloud      2026-06-10    21d     1/1     GAP wrong_scope         READY   <-
A1.2-b    A1.2      cloud      2026-06-30     1d     6/6     READY                   READY

The existence-only arm is not a straw man. It is what an evidence tracker built around collection status does, and all six deficiencies have an artifact that exists, hashes correctly, and was collected without error.

The six planted deficiencies, and why each is subtle

Each one is a different failure and produces a different gap reason, because "you have a gap" is not actionable and the six get fixed in six different places.

A1.2-a deserves a note: the misconfiguration is left in controls/soc2.yaml, marked as deliberate. Somebody wiring a collector at the wrong account is one of the most common things that actually happens on an engagement, and it is the failure provenance exists to catch.

Credential revocation: the silent kind

Four scenarios, one per system, split across the two ways a real API reacts to a narrowed credential. A silent gap is a control a checker still calls audit-ready while its evidence is degraded — computed from the artifacts, not from the checker's own verdict, so neither arm is grading itself.

The two deny scenarios are the easy half; any tool notices a 403. The filter ones are the point:

GAP CC6.1-a (collection_failed): observed 0 of 62 expected records — an empty result
cannot support a compliance claim, whether the population is truly clean or the
credential can no longer see it

That artifact's content, read on its own, says users_examined: 0, users_without_mfa: [], exception_count: 0. There is nothing in the payload to notice. The only thing that catches it is knowing the workforce is 62, from a directory:stats endpoint on a different scope. Five of the fifteen controls have that kind of independent population cross-check, and they are exactly the endpoints that can silently return fewer rows — that is the design rule, and a test pins which scopes are the silent ones so an endpoint cannot quietly switch to raising and make these scenarios pass on the easy path.

A failed collection still produces a ledgered artifact with status='failed' and the reason. "We could not collect this" is evidence an auditor needs, and logging the error and continuing is what loses it.

Provenance: hashed, chained, and attacked

Each artifact binds its content with sha256. An append-only ledger binds the artifact id, the content digest, and a digest of the provenance — control, system, scope, collecting identity, both timestamps, population counts, status — to the previous entry's hash. Content hashing alone would catch an edited artifact but not a deleted one, and an evidence package with a quietly removed inconvenient artifact is the failure that actually matters in an audit.

Eight tamper classes, each applied to a throwaway copy of the real database, each asserting which detector fired:

The two middle rows are the ones with a motive: each is a single-field edit that would turn a real gap green, and each is caught because provenance is hashed into the ledger rather than merely stored beside the content. The untampered store verifying clean is checked first — a detector that fires on an intact package would "catch" all eight and mean nothing.

The auditor disagrees in the right places

An independent reviewer grades artifact quality on five criteria. It does not import classify, effectiveness, or provenance, and it recomputes every hash with its own hashlib and json calls, so agreement between the two is a signal rather than a tautology. Both facts are enforced by tests that walk the import graph.

                        provenance  freshness  population  scope  attested   verdict
9 audit-ready controls      ok          ok         ok        ok      ok      accept
CC6.3                       ok          NO         ok        ok      ok      reject
CC6.7                       ok          ok         NO        ok      ok      reject
A1.2-a                      ok          ok         ok        NO      ok      reject
CC7.4                       ok          ok         ok        ok      NO      reject
CC7.1, CC8.1-a              ok          ok         ok        ok      ok      accept  <-

Two results, and the second is the interesting one:

• Every control this pipeline calls audit-ready passes independent review, 9 of 9. Without that, "audit-ready" is a label the tool grants itself.

• CC7.1 and CC8.1-a are accepted on quality and still refused by the pipeline. Their evidence genuinely is fine; their controls are failing. An auditor that rejected all six planted deficiencies would agree with the pipeline for the wrong reason, and CI fails if that separation ever collapses.

Each rejection also names a single specific criterion, tested — a generic rejection is not a finding anyone can act on.

Collectors are MCP tools, four servers, one credential each

evidence mcp-tools

Four servers, not one. Each holds exactly one system's credential and publishes only that system's collectors, so asking the CI server for identity evidence is refused by name:

CC6.1-a draws its evidence from 'idp'; this server only holds the 'ci' credential.
Ask the 'idp' server.

Every tool returns the evidence plus the provenance the store needs — the scope actually read, effective_at, observed and expected population and where the expectation came from, and a status of collected or failed. A tool that reports failed has not given you evidence, and the published description says so.

This is not a parallel implementation. evidence collect --via-mcp drives the entire collection run through the tools, and a test asserts the resulting artifacts are byte-identical to the in-process path — same content hashes, same provenance, same population counts. An MCP server that is built but never driven end to end can drift from the code it claims to expose, and nobody notices until an agent gets a different answer than the dashboard.

Scheduling: stale evidence stays due

Due-ness is computed from effective_at, which has a consequence worth stating: a control whose only evidence is a 415-day-old attestation stays due forever, no matter how many times the agent successfully re-fetches it. Re-collecting stale evidence is not progress. A scheduler keyed on collection timestamps would mark CC6.3 done every quarter and never surface it again.

uv run evidence collect --only-due   # collects CC6.3 and skips the other 14

Reproduce

uv sync --extra dev
uv run evidence init                 # collect the period + replay the review queue

uv run pytest                        # 172 tests
uv run evidence dashboard --compare  # both checkers, side by side
uv run evidence gaps                 # scored against the planted deficiencies
uv run evidence verify               # provenance over the whole ledger
uv run evidence tamper-demo          # 8 tamper classes, 8 expected detectors
uv run evidence audit                # the independent artifact-quality review
uv run evidence inject all           # revoke each credential, count silent gaps
uv run evidence review               # the human queue for judgment controls
uv run evidence mcp-tools            # the published collection surface
uv run evidence collect --via-mcp    # the same run, over the protocol
uv run evidence show CC6.7 --content # one control, end to end
uv run evidence report               # writes results/report.json + dashboard.md

python scripts/mutation_check.py     # break each guard; all 20 must be caught
python scripts/ci_local.py           # every CI step, parsed from the workflow YAML

Everything runs offline: no model, no network, no secrets. EVIDENCE_AS_OF moves the collection date if you want to watch evidence age out of its windows.

Layout

4.07-compliance-evidence-agent/
├── controls/
│   ├── soc2.yaml            # 15 controls -> evidence requirements, as configuration
│   └── signoffs.yaml        # this cycle's recorded human review decisions
├── src/evidence/
│   ├── systems/             # the four company stand-ins
│   │   ├── base.py          # credentials, scopes, and DENY vs FILTER (enforced)
│   │   ├── idp.py           # workforce, admins, provisioning, access review, devices
│   │   ├── cloud.py         # two accounts: inventory, network, logs, backup, alarms
│   │   ├── ci.py            # merged changes, branch protection, vulnerability scans
│   │   └── ticketing.py     # incidents and the IR tabletop record
│   ├── controls.py          # loads and hard-validates the framework
│   ├── collectors.py        # one per evidence kind, plus the population guard
│   ├── mcp_server.py        # the collectors as scoped MCP tools, per system
│   ├── gateway.py           # in-process and over-MCP paths, proven equivalent
│   ├── artifacts.py         # content hashing, provenance, the append-only ledger
│   ├── provenance.py        # verification: recompute everything, trust nothing
│   ├── collection.py        # the schedule, and the collection run
│   ├── effectiveness.py     # is the control OPERATING, as opposed to documented
│   ├── classify.py          # audit-ready or not, and exactly why; both arms
│   ├── review.py            # the human queue, and independence
│   ├── auditor.py           # independent artifact-quality review
│   ├── tamper.py            # the eight attacks on the evidence package
│   ├── inject.py            # credential revocation, and silent-gap scoring
│   ├── groundtruth.py       # the planted deficiencies (unreachable from detection)
│   ├── report.py            # assembles results/
│   └── cli.py               # evidence init|collect|dashboard|verify|audit|inject|...
├── scripts/
│   ├── mutation_check.py    # breaks each guard; every mutant must be caught
│   └── ci_local.py          # runs CI's steps verbatim from the YAML
├── results/                 # committed evidence (report, dashboard, mutation check)
└── data/                    # the evidence store (gitignored; rebuilt by `evidence init`)

Honesty notes

• The four systems are stand-ins, and say so. They are shaped like an IdP, a cloud account, a CI system, and a ticketing system — same pagination, same scope model, same two failure behaviours — but they are in-process Python, not real APIs. That is a deliberate trade: labelled compliance deficiencies at scale do not exist to test against, and a planted corpus is the only way to measure gap detection honestly.

• 1.00 precision and recall are on a corpus I designed. The meaningful claim is not the number, it is that all six deficiencies are invisible to an existence-only check and each guard is independently load-bearing — the mutation check removes any one of the twenty and a specific test fails. On real data these numbers would be lower; the method is the transferable part.

• min_coverage: 1.0 is a strict choice. Requiring the whole population makes the truncated scan a gap. A real engagement might accept a sampling threshold; that is a config change per control, and the point is that the comparison happens at all.

• Only 5 of 15 controls have an independent population cross-check. The other ten are single records or complete lists with no separate counter, so their expectation is the minimum the control requires. population_source records which is which, and the auditor reports it, rather than the tool implying all fifteen are cross-checked.

• The auditor shares controls with the pipeline, on purpose. An auditor is handed the framework; reading a different definition of the requirements would make any disagreement meaningless. It shares nothing else, and a test enforces that.

• Effectiveness rules are simple by design. They are readable predicates over the collected evidence, not a model. That is the right shape for a control that has to be explained to an auditor, and it means the rules are only as good as the evidence fed to them — which is why most of this project is about the evidence.

• Nothing here writes to a real system. Every collector is read-only, the systems are in-process, and no code path reaches an external API.

• Every number in this README is machine-generated. All of them come from results/report.json, written by evidence report, and CI regenerates and gates on it.

Bugs this build found in itself

Kept because each one would have produced a confidently wrong deliverable.