DataOps MCP Copilot

An auditable copilot for data operations. It answers operational questions over a synthetic company environment (issue tracker + pipeline warehouse) through MCP tool servers, cites every claim to the tool call it came from, and refuses to change anything without human approval.

Everything in this repo is synthetic: the company, the people, the pipelines, the incidents. There is no real employer data, code, or schema here. See docs/CLEAN_ROOM.md.

What it does

• Two MCP servers expose the environment as tools: tickets (search, read, propose changes) and warehouse (pipeline health, runs, incidents, governed metrics).

• An agent loop drives those tools to answer questions. Every tool result is numbered, and the final answer cites those numbers, so any claim can be traced back to the exact output it came from.

• Writes are proposal-only. The agent can propose a ticket update or comment; the proposal gets a token and sits in an approvals queue until a person approves it. The apply path refuses anything that was not approved first, and every step lands in an audit log.

• A small web UI lets you ask questions, inspect the full trace (tool calls, arguments, results, latency), and work the approvals queue.

The seeded environment has a few stories to find: a deploy that broke a reconciliation job, a slow-burn latency creep, and a sev1 outage with follow-up tickets.

Related MCP server: Sentinel MCP Server

Requirements

Python 3.10+. SQLite by default; Postgres works by setting DATABASE_URL. An OpenAI key is optional (needed only for the live model; the mock provider runs everything offline).

Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m dataops.seed        # build the synthetic environment

Run it

Web UI (ask, traces, approvals):

uvicorn dataops.api:app --port 8000

CLI:

python -m dataops.agent "Why does billing_reconcile keep failing?"

By default the copilot uses the mock provider: a scripted, deterministic planner that needs no API key. To run against a live model:

export OPENAI_API_KEY=...            # or drop it in .secrets/openai_key
COPILOT_PROVIDER=openai uvicorn dataops.api:app

Evaluation

python -m evals.run_eval --provider mock --write-report

25 fixed operational questions against a freshly seeded environment, scored on tool selection, grounded facts, and a no-mutation-without-approval safety check. Current results: 25/25 with the mock provider (deterministic, in CI); the same set passes with gpt-4o-mini (about $0.05 per full run). Details in docs/EVALUATION.md.

Tests

python -m pytest tests/

Layout

dataops/
  seed.py            deterministic synthetic environment (fixed RNG seed)
  models.py          schema: pipelines, runs, incidents, tickets, approvals, audit
  servers/           the two MCP servers (stdio)
  client.py          MCP hub: launches servers, namespaces tools
  agent.py           the copilot loop (tools -> cited answer)
  guardrails.py      the write gate: propose -> human approves -> apply
  tracing.py         per-question JSONL traces
  providers/         openai (live) + mock (offline, deterministic)
  api.py             FastAPI app
  static/index.html  the UI, no build step
evals/               question set + scoring harness
tests/               pytest suite
docs/                architecture, design decisions, clean-room note, eval results

Notes

• The mock provider exists so reviewers can run everything without a key and CI stays deterministic. It is a scripted planner, not a model; the README calls that out rather than hiding it.

• LLM-facing tools validate their enums and return corrective hints ("'urgent' is a severity, pass it as severity"). Models recover from these instead of concluding nothing exists.

• The agent has a hard step ceiling; it would rather stop and say so than burn budget looping.