# DataOps MCP Copilot

An auditable copilot for data operations. It answers operational questions over
a synthetic company environment (issue tracker + pipeline warehouse) through
MCP tool servers, cites every claim to the tool call it came from, and refuses
to change anything without human approval.

Everything in this repo is synthetic: the company, the people, the pipelines,
the incidents. There is no real employer data, code, or schema here. See
docs/CLEAN_ROOM.md.

## What it does

- Two MCP servers expose the environment as tools: `tickets` (search, read,
  propose changes) and `warehouse` (pipeline health, runs, incidents,
  governed metrics).
- An agent loop drives those tools to answer questions. Every tool result is
  numbered, and the final answer cites those numbers, so any claim can be
  traced back to the exact output it came from.
- Writes are proposal-only. The agent can propose a ticket update or comment;
  the proposal gets a token and sits in an approvals queue until a person
  approves it. The apply path refuses anything that was not approved first,
  and every step lands in an audit log.
- A small web UI lets you ask questions, inspect the full trace (tool calls,
  arguments, results, latency), and work the approvals queue.

The seeded environment has a few stories to find: a deploy that broke a
reconciliation job, a slow-burn latency creep, and a sev1 outage with
follow-up tickets.

## Requirements

Python 3.10+. SQLite by default; Postgres works by setting `DATABASE_URL`.
An OpenAI key is optional (needed only for the live model; the mock provider
runs everything offline).

## Setup

    python -m venv .venv && source .venv/bin/activate
    pip install -r requirements.txt
    python -m dataops.seed        # build the synthetic environment

## Run it

Web UI (ask, traces, approvals):

    uvicorn dataops.api:app --port 8000

CLI:

    python -m dataops.agent "Why does billing_reconcile keep failing?"

By default the copilot uses the mock provider: a scripted, deterministic
planner that needs no API key. To run against a live model:

    export OPENAI_API_KEY=...            # or drop it in .secrets/openai_key
    COPILOT_PROVIDER=openai uvicorn dataops.api:app

## Evaluation

    python -m evals.run_eval --provider mock --write-report

25 fixed operational questions against a freshly seeded environment, scored on
tool selection, grounded facts, and a no-mutation-without-approval safety
check. Current results: 25/25 with the mock provider (deterministic, in CI);
the same set passes with gpt-4o-mini (about $0.05 per full run). Details in
docs/EVALUATION.md.

## Tests

    python -m pytest tests/

## Layout

    dataops/
      seed.py            deterministic synthetic environment (fixed RNG seed)
      models.py          schema: pipelines, runs, incidents, tickets, approvals, audit
      servers/           the two MCP servers (stdio)
      client.py          MCP hub: launches servers, namespaces tools
      agent.py           the copilot loop (tools -> cited answer)
      guardrails.py      the write gate: propose -> human approves -> apply
      tracing.py         per-question JSONL traces
      providers/         openai (live) + mock (offline, deterministic)
      api.py             FastAPI app
      static/index.html  the UI, no build step
    evals/               question set + scoring harness
    tests/               pytest suite
    docs/                architecture, design decisions, clean-room note, eval results

## Notes

- The mock provider exists so reviewers can run everything without a key and
  CI stays deterministic. It is a scripted planner, not a model; the README
  calls that out rather than hiding it.
- LLM-facing tools validate their enums and return corrective hints ("'urgent'
  is a severity, pass it as severity"). Models recover from these instead of
  concluding nothing exists.
- The agent has a hard step ceiling; it would rather stop and say so than
  burn budget looping.