OpsPilot — Enterprise AI Workflow Agent

Fictional SaaS customer support investigation agent, using FastAPI, SQLite/SQLAlchemy, Alembic, MCP, LangGraph, optional local Ollama inference, and a dependency-free web dashboard.

Status: Milestone 5 complete (version 0.5.0), plus the Milestone 6A offline evaluation benchmark (Offline evaluation). The dashboard, investigation API, CLI and MCP tools work end to end on a local machine with a local Ollama model and the seeded SQLite database. The project is not production-ready: see Production deployment limitations.

Run locally on Windows (PowerShell)

From the repository root:

.venv\Scripts\python.exe -m pip install --no-cache-dir -e ".[dev,agent]"
.venv\Scripts\python.exe -m alembic upgrade head
.venv\Scripts\python.exe -m enterprise_ai_workflow_agent.db.seed
.venv\Scripts\python.exe -m pytest -q
.venv\Scripts\python.exe -m ruff check .
.venv\Scripts\python.exe -m alembic check
.venv\Scripts\python.exe scripts\smoke_mcp.py

Install Ollama separately from https://ollama.com/download and pull a model:

ollama pull qwen2.5:3b
ollama list

Start the Ollama service if it is not already running (ollama serve in a separate terminal).

Dashboard

Start the server from the repository root (the default DATABASE_URL, sqlite:///./opspilot.db, is relative to the working directory, and the MCP tool server inherits it):

.venv\Scripts\python.exe -m uvicorn enterprise_ai_workflow_agent.main:app --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000/. Pick an example or type a question, choose the number of tool rounds and select Investigate (or press Ctrl+Enter). The result shows:

• Answer: a short narrative and where it came from (model, tool-less synthesis, or a deterministic evidence summary), with a warning banner for limited investigations.

• What the evidence supports: rule-generated sections (relevant tickets, relevant documents, other records, unknowns). Record lists and counts come only from here.

• Retrieved records: every record returned by the tools, with its relevance label.

• Tool execution timeline: each call with its arguments, adjustments, errors, and calls added by the orchestrator.

Stop waiting cancels the request. The server notices the closed connection within about a second and cancels the investigation, which shuts down its MCP subprocess and frees the concurrency slot. API docs are at http://127.0.0.1:8000/docs.

Investigation API

POST /api/v1/investigations runs one investigation synchronously and returns the answer, evidence, analysis sections and tool trace:

$body = '{"question": "Investigate SSO issues for Northstar Labs. Check tickets and documentation.", "max_rounds": 4}'
Invoke-RestMethod http://127.0.0.1:8000/api/v1/investigations -Method Post -ContentType application/json -Body $body

• Request: question (1-4000 characters, required), optional model (an installed Ollama tag) and max_rounds (1-8). Unknown fields are rejected.

• 200: status is completed or limited (with limited_reason). Other fields: answer, narrative, answer_source, answer_notice, round_limit_reached, tool_rounds, tool_calls, duration_seconds, tool_trace, evidence, analysis.

• Errors use one shape, {"id", "question", "status": "failed", "error": {"code", "message", "details"}, "duration_seconds"}: 422 invalid_request, 404 model_unavailable, 503 ollama_unavailable / ollama_integration_missing / mcp_unavailable / model_inference_failed, 504 timeout, 500 internal_error. Messages never include configuration values, paths or exception details.

• GET /api/v1/investigations/config returns the default model, round limits, question length limit and timeout (no URLs or credentials).

Runtime settings (environment variables or .env, see .env.example): OLLAMA_BASE_URL, OLLAMA_MODEL, INVESTIGATION_TIMEOUT_SECONDS (default 300, covers queueing too), INVESTIGATION_MAX_ROUNDS (default 4), INVESTIGATION_MAX_CONCURRENCY (default 1; further requests queue).

CLI

.venv\Scripts\python.exe -m enterprise_ai_workflow_agent.agent.cli "Investigate SSO issues for Northstar Labs. Check support tickets and documentation. Cite the evidence." --model qwen2.5:3b

qwen2.5:3b is a small starting model; tool-calling accuracy and response quality vary. If resources allow, try a larger tool-capable model. Ollama model downloads can take multiple GB of disk space. Confirm free space before pulling weights.

Related MCP server: Archive MCP Server

Offline evaluation (Milestone 6A)

.venv\Scripts\python.exe -m enterprise_ai_workflow_agent.evaluation              # all cases
.venv\Scripts\python.exe -m enterprise_ai_workflow_agent.evaluation --case U3     # one case

Runs the versioned benchmark evals/benchmark_v1.json (25 cases: 6 successful investigations, 4 missing accounts, 5 incomplete evidence, 7 unsafe conclusions, 3 tool-discipline cases) and writes evals/results/benchmark_v1-offline.{json,md} (git-ignored). Runs filtered with --case write ...-offline-subset-<IDs>.{json,md} and never overwrite the full-suite report. Reports record the benchmark's content SHA-256 and which cases ran. The exit code is 1 if any case fails; failures print the case ID, expected vs observed values and the tool trace. A case that raises is reported as a failure (run_completed) and the remaining cases still run. The same cases run in pytest (tests/test_evaluation.py), and every case must pass.

Each case runs the real LangGraph agent against the real MCP server in memory on its own SQLite fixture with explicit record IDs (evaluation/harness.py; production seeding is not used). The model is scripted: tool-loop and synthesis replies come from the case. No Ollama, network or subprocess is needed, and reports contain no timings, so repeated runs are identical. A script that runs out of replies, or has replies left over, fails the case, so a case cannot pass on a graph path that never requested its scripted text.

The benchmark covers pre-resolved and model-first (pre_resolve: false) runs, multi-page ticket retrieval, rejected guessed IDs and skipped duplicate calls, failed lookups, the round limit, tool-loop and synthesis failures, and seven unsupported-conclusion guards. tests/test_evaluation.py disables each of those guards in turn (mutation tests) and checks that the corresponding case fails, and checks that replacing a synthesis with generic text fails every case whose answer must come from synthesis.

Metrics, per case and aggregated per category (evaluation/metrics.py):

Expectations accept equivalent wording (each fact lists alternative regexes) and use upper bounds rather than exact call sequences. Limitations:

• Scripted replies test orchestration, retrieval and answer guardrails, not model quality. Live-model evaluation (repeated runs against Ollama) is future work.

• All checks are lexical. They detect the specific claims each case names; they are not general hallucination detection. A wrong statement that no pattern covers goes unnoticed, and a narrative fact only shows that the expected wording is present, not that the rest of the narrative is true.

• Citation parsing is heuristic: a typeless #N that matches records of several types is reported as ambiguous and not scored; IDs written without # or a record type ("see 101") are not recognised; a number listed after an ID ("Ticket 101, 2 users") is read as a second ID.

• The unsupported-claim metric reuses the agent's own grounding detectors, so it guards against regressions but is not an independent judge.

• Runs patch module-level MCP server state (restored on exit, including after exceptions) and must not run concurrently.

Add a case by appending it to evals/benchmark_v1.json (the schema is in evaluation/dataset.py). Use --verbose to see agent logs. Changing the meaning of existing cases calls for a new benchmark version rather than an edit.

Architecture

Browser dashboard (or CLI) -> FastAPI POST /api/v1/investigations -> InvestigationService (concurrency limit, timeout, cancellation; one worker thread and event loop per run) -> LangGraph StateGraph with an Ollama chat model -> MCP stdio client -> MCP server subprocess (one per investigation) with five read-only tools -> SQLAlchemy -> SQLite. The result passes deterministic grounding checks before it is returned. The data endpoints (/api/v1/accounts, /api/v1/tickets, ...) read the database directly.

Tool calls are limited to 4 rounds by default, with at most 5 tool calls per round; tool output is truncated to 12,000 characters per call. Unknown tool names are rejected. Errors become tool messages for the agent to interpret. The final response includes a tool-call trace. There is no persistent memory across runs.

Each investigation's MCP subprocess is closed when the run ends, fails, times out or is cancelled (on Windows through a job object, so the venv launcher's child process exits too). Uvicorn does not cancel a request handler when the client disconnects, so the endpoint polls for a closed connection and cancels the run itself.

Reliability safeguards for small models

• Bounded arguments. Tool schemas publish minimum/maximum for limit (1-20). Out-of-range numbers are clamped to the schema bounds before the MCP call; the trace keeps the original and sent arguments plus an adjustments note. Record IDs are never changed. The MCP server still validates every call.

• Clean tool errors. Validation errors reach the model as short, actionable text (for example Invalid arguments. limit: Input should be less than or equal to 20). Unexpected server exceptions are logged and returned as a generic message.

• Intent-aware follow-ups. When the model stops, deterministic rules (no extra LLM call) check whether evidence the question explicitly asks for is missing: support-issue terms (ticket, incident, error, failure, ...) require tickets for the verified account the question names; guidance terms (troubleshoot, documentation, how to, next steps, ...) require a knowledge search. Missing lookups run through the same validated tool path, within max_rounds, and appear in the trace with origin: "orchestrator" and a reason. Dependent lookups chain (an account search, then its tickets) while rounds remain. Account-only questions get no ticket lookups, and several matching accounts are never guessed between.

• Account discovery. If a ticket or account question ends with no account lookup at all (in testing, qwen2.5:3b sometimes returns an empty first reply), and exactly one account name can be read from the question (a run of capitalised words that are not question words or acronyms such as SSO), search_accounts is called with that text. Account IDs still come only from the MCP result. Several candidate names, or none, make the investigation limited with an explanation; nothing is guessed. If the model's own account search matched nothing but did not search a name from the question on its own (for example both names in one query), each such name is searched separately; an empty result is a negative finding only for names that were actually searched.

• Round budget. The last tool round goes to required lookups first; the model's own calls in that round run alongside them within the per-round call limit. max_rounds is never raised.

• Repeated lookups. An identical call that already succeeded is not re-executed. If a model turn only repeats earlier lookups while required evidence is missing, the orchestrator uses that round for the missing lookup instead.

• Transient model failures. Ollama "token repeat" errors, empty streams and empty replies (no text and no tool call) are retried within a fixed budget of 4 attempts. Persistent empty replies are treated as the model stopping, not as a failure.

• Answer = model narrative + deterministic sections. Record lists and counts never come from the model. Every answer ends with sections rendered from the labelled records: Relevant support tickets, Relevant knowledge documents (or an explicit statement that documentation-backed guidance is unavailable), Other retrieved records and Unknown or unconfirmed. The dashboard shows the same sections. The model writes only a short explanatory narrative.

• Final synthesis. Evidence collection and narrative writing are separate. Once the required evidence is in hand and the model stops (or the orchestrator completed the evidence, or the tool-round limit is reached), a single synthesis call, made with the model without tools, writes the narrative. It sees only the question and the verified records, grouped by relevance with their IDs. It also replaces a tool-loop answer that fails the grounding checks. Synthesis runs at most once (transient errors are retried within the attempt budget), stays inside the investigation timeout, and is skipped when required evidence is missing.

• Grounding for every narrative. The tool-loop answer, a synthesis, and a synthesis after the round limit all pass the same checks: cited IDs must exist with the same record type; records without a topic match must not be presented as related; record counts must match the records (a count of "related" tickets must equal the relevant ones); troubleshooting instructions must cite a relevant retrieved document, otherwise they are removed with a visible note; causal claims get an "unconfirmed" note. A narrative that fails is replaced by a deterministic one (answer_source: "evidence_summary" with an answer_notice), never an invented conclusion.

• Fix claims. No record confirms that a step resolves a customer's problem, so present- or future-tense claims that one does ("checking the settings resolves the issue", "this will fix the login failures") are softened to "may help resolve/fix", with a visible note. Ticket status wording ("Ticket 2 was resolved", "is resolved"), hedged, negated and quoted text are left unchanged. The synthesis prompt also asks the model not to make such claims.

• Status. completed: the required evidence was retrieved and a grounded answer was returned. limited: required evidence is still missing (lookup failed, ambiguous or unidentifiable account, or the round budget ran out first), explained in limited_reason. Reaching the tool-round limit alone is reported as round_limit_reached, not as limited. Retrieving records that do not match the question, or finding no matching account, is a complete, negative result (the answer states which search matched nothing). Infrastructure failures in the tool loop are failed errors.

• Relevance. Records are labelled by lexical overlap with the question's topic, plus a few explainable groups of related terms (for example sso, saml and login for sign-in problems). Generic symptom words (failure, error, ...) alone do not make a record relevant when the question names a topic. Each relevant record carries a relevance_basis. A related topic does not establish a cause.

Production deployment limitations

The application is built and tested for local, single-user use. Before any shared or production deployment it would still need:

• Security: no authentication, authorization, tenant isolation or rate limiting. Bind the server to 127.0.0.1 (as above) and do not expose it publicly.

• Data: synthetic seed data only; tools are read-only and there is no approval system.

• Scaling: the concurrency limit and request queue live in one server process; several Uvicorn workers would each apply their own limit.

• Operations: no observability backend (the trace is per run), and the application's INFO logs are not shown under Uvicorn's default logging configuration.

• Testing: CI uses fake LLM/MCP clients (no Ollama or model download). Real-model and browser checks are run manually.

Docker (current limitations)

The Dockerfile builds and starts the API and dashboard, and CI builds the image, but the image is not yet a working investigation environment (planned for Milestone 8):

• It installs the package without the agent extra, so investigations return 503 ollama_integration_missing.

• OLLAMA_BASE_URL defaults to 127.0.0.1, which inside a container is the container itself. Ollama on the host needs host.docker.internal or a shared network.

• The image contains neither migrations//alembic.ini nor a database, and nothing creates tables at startup, so the data endpoints and tools fail unless a prepared SQLite file is mounted and DATABASE_URL points to it.

• There is no compose file, health check or Ollama service definition.

Answer accuracy limitations

• Evidence grounding is rule-based and prompted, not formally verified. Check citations against the tool output. There is no guarantee of factual correctness or of immunity to prompt injection.

• Knowledge search uses lexical token overlap, not embeddings/vector retrieval.

• The guards check record IDs and types, relevance claims, record counts, uncited troubleshooting instructions and causal wording. They do not check the model's interpretation: a narrative may still overstate (for example, calling a resolved ticket "the solution"). The deterministic sections below the narrative are the authoritative record list.

• Troubleshooting advice is detected by phrasing (imperatives, "you should", "next steps"...). Softer wording such as "it is important to address this" is not removed.

• Fix claims are detected by phrasing too: "X resolves/fixes/will fix ..." is softened, but paraphrases such as "is the solution" or "addresses the root problem", and past-tense claims ("this resolved the issue"), are not changed.

• Account discovery reads capitalised names. Lowercase names, or a single-word name at the start of a sentence, are not recovered. A capitalised product name next to an account name makes the question ambiguous, and the run ends limited instead.

• Relevance is lexical, with a few hand-written groups of related terms; it is not semantic search.

• qwen2.5:3b often spends tool rounds on invented IDs (blocked) or repeated calls. Real investigations took about 12-160 s on a laptop (most 30-90 s); a question with no identifiable account can use the whole round budget before it ends limited.

• Reliability figures come from small manual batches (8 real runs each), not a benchmark.

Existing MCP server

.venv\Scripts\python.exe scripts\smoke_mcp.py

The server supports search_accounts, get_account_tickets, get_ticket_details, get_product_information, and search_knowledge.