platform-support-agent

An internal platform-support agent for Meridian Cloud, a fictional SaaS platform company. An employee or support engineer pastes a ticket — VPN, deploy failure, 5xx, quota — and the agent retrieves help-center and runbook passages with citations, calls permissioned tools over MCP, and then answers, escalates, or proposes an action.

It will not restart anything without the sre role and a human pressing confirm.

Everything here is synthetic. No real company data, tickets, or customer names.

git clone https://github.com/ninja-nb/platform-support-agent && cd platform-support-agent
make setup && make index
make eval          # golden set, offline, no API key needed
make ui            # support console on :8501

Why this shape

Three constraints drove the design, and they are the ones that actually decide whether a support agent can be turned on for real users.

A wrong answer costs more than no answer. The agent answers only from retrieved passages and cites every claim. When retrieval comes back empty it says so and offers to file a ticket. Three of the twenty golden cases exist purely to catch confident answers to questions the corpus cannot support.

Stale documentation is the default state of a help center. Deprecated documents are not deleted from the index — users quote them, so they have to be findable. Instead a superseding document is structurally guaranteed to rank above the document it replaces, and the agent is required to name the supersession rather than follow retired steps. A relevance penalty alone could not promise this: a retired runbook is often the single best lexical match, precisely because the user is reading from it.

Read access and write access are different risks. Retrieval and status reads are open to everyone. State-changing actions are role-gated, need a human confirmation round-trip, and are recorded in an append-only audit log. The confirmation token is bound to the exact service, environment, and user, so it cannot be replayed against a different target.

Being authorized and being correct are different questions. err-005 says a restart only applies to a wedged service, and that restarting a saturated one makes the outage worse by removing the capacity still serving. That rule is enforced in the tool layer, not left to the model: an sre who insists on restarting a healthy-but-saturated service is refused and pointed at scaling out. Guardrails that depend on the model reasoning correctly are not guardrails.

Related MCP server: zendesk-mcp

What it does

Current numbers are in docs/EVALS.md, regenerated by make eval.

The baseline is deliberately not impressive. The default provider is stub: a rule-based planner with no model behind it, which exists so the harness runs offline in CI and so the golden set has a published control. It passes 13 of 20 cases, with safety at 5 of 6. That is the bar a real provider has to clear, and the seven failures are the work queue — they are listed in docs/EVALS.md rather than smoothed away.

Read the regression log first if you only read one thing. Three entries so far, including one finding that a refusal threshold cannot be tuned to separate an unanswerable question from an answerable one on a lexical retriever, with the measurements that show why.

How it fits together

ticket text
    |
    v
agent loop  --------> provider (stub | openai | vertex)
    |                     returns: tool calls, or a final answer
    v
call_tool  <-- the only entry point to the tool surface
    |  authorize (roles.py) -> confirmation gate -> audit log -> execute
    v
search_docs   lookup_ticket   get_status   get_deploys   create_ticket   restart_service
    |
    v
answer + citations + behavior label  ---> eval harness scores it

The one design decision worth calling out: tools are plain Python functions behind a single call_tool dispatcher, and MCP is a transport in front of that dispatcher, not the place the rules live. The agent, the eval suite, and an external MCP client all traverse the same authorization code. Putting the permission checks in the MCP layer would have meant the eval suite tested a different code path than production used.

Roles come from the server environment, never from the client. A client that could name its own role would make the permission table decorative.

What the agent must do is in docs/PRD.md, stated as behaviours with the eval case that owns each one. How it is built is in docs/ARCHITECTURE.md, with one record per decision in docs/adr/ — including the six decisions not yet made. What an adversary can do to it is in docs/THREAT_MODEL.md.

Tool permissions

Evaluation

The eval harness is the part of this repo worth reading first.

Each case declares assertions independently — which documents must be cited, which must not be, which tools are required or forbidden, what the response text must contain, and which behaviour label is acceptable. A regression therefore names the property that broke instead of just turning a case red.

Assertions roll up into four reported families (groundedness, correct-tool, behaviour, content) plus one gate:

Safety is a gate, not an average. The permission, unsafe-restart, and unanswerable categories ship at 100% or they do not ship. A groundedness average of 94% that hides a permission bypass is worse than useless, so --require-safety scores those categories separately and fails the build on its own.

Case categories: how_to, stale_doc, wrong_diagnosis, incident_lookup, missing_evidence, wrong_service, permission_denied, unsafe_restart, unanswerable, cross_doc, escalate.

Two categories deserve explanation, because they are what keep the numbers honest when both the corpus and the questions are synthetic:

• unanswerable asks plausible questions the corpus cannot answer. Without these, a groundedness score measures nothing but the retriever's willingness to return something.

• wrong_diagnosis pairs INSUFFICIENT_CAPACITY against QUOTA_EXCEEDED. They look alike and have opposite resolutions — retrying a different zone fixes one and can never fix the other. Confusing them is the most likely real-world failure.

data/evals/holdout.jsonl is never tuned against. It exists so that a golden-set score climbing over four weeks can be checked against something that was not used to get there.

make eval                                   # golden set, writes docs/EVALS.md
psa-eval --file holdout.jsonl               # held out; do not tune against this
psa-eval --require-safety --fail-under 0.8  # CI gates

Layout

data/corpus/        7 synthetic help-center articles (one deliberately deprecated)
data/fixtures/      fake tickets, environments, deploy history
data/evals/         golden.jsonl (20 cases), holdout.jsonl (4)
src/psa/roles.py    the permission table: the security boundary, one file, no deps
src/psa/rag/        chunking, BM25 index, Retriever seam
src/psa/mcp_server/ tools, call_tool dispatcher, audit log, MCP stdio transport
src/psa/agent/      loop, system prompt, behaviour policy
src/psa/providers/  provider interface; stub / openai / vertex
src/psa/evals/      scoring and the runner
ui/app.py           Streamlit support console

The core — retrieval, permissions, tools, eval harness — is stdlib-only on purpose, so make eval works on a fresh clone with no API key and no network. Model SDKs are optional extras.

Roadmap

Honest status: weeks 2 through 4 are not done.

• Week 2. Run the OpenAI provider against the live API and record its scores beside the stub baseline. Port the loop to LangGraph for session memory and resumable confirmation. Close the last safety case (see regression log R2).

• Week 3. Vertex/Gemini skin, and swap BM25 for embeddings behind the existing Retriever interface so the golden set measures whether it actually helped.

• Week 4. Write up one eval miss end to end — symptom, root cause, fix, score movement — in the regression log. Record p95 latency and cost per request.

• Week 5, if it earns it. Vertex Vector Search, Cloud Run, an ADK skill wrapper.

Non-goals

Not a coding agent. Not an SRE incident copilot — incident triage is one ticket type here, not the product. No real customer data, ever. No fine-tuning: the interesting problems in this shape of system are retrieval quality, permissions, and evaluation, and none of them are solved by training a model.