# platform-support-agent

An internal platform-support agent for **Meridian Cloud**, a fictional SaaS platform
company. An employee or support engineer pastes a ticket — VPN, deploy failure, 5xx,
quota — and the agent retrieves help-center and runbook passages with citations, calls
permissioned tools over MCP, and then answers, escalates, or *proposes* an action.

It will not restart anything without the `sre` role and a human pressing confirm.

Everything here is synthetic. No real company data, tickets, or customer names.

```bash
git clone https://github.com/ninja-nb/platform-support-agent && cd platform-support-agent
make setup && make index
make eval          # golden set, offline, no API key needed
make ui            # support console on :8501
```

---

## Why this shape

Three constraints drove the design, and they are the ones that actually decide whether
a support agent can be turned on for real users.

**A wrong answer costs more than no answer.** The agent answers only from retrieved
passages and cites every claim. When retrieval comes back empty it says so and offers
to file a ticket. Three of the twenty golden cases exist purely to catch confident
answers to questions the corpus cannot support.

**Stale documentation is the default state of a help center.** Deprecated documents are
not deleted from the index — users quote them, so they have to be findable. Instead a
superseding document is *structurally guaranteed* to rank above the document it
replaces, and the agent is required to name the supersession rather than follow retired
steps. A relevance penalty alone could not promise this: a retired runbook is often the
single best lexical match, precisely because the user is reading from it.

**Read access and write access are different risks.** Retrieval and status reads are
open to everyone. State-changing actions are role-gated, need a human confirmation
round-trip, and are recorded in an append-only audit log. The confirmation token is
bound to the exact service, environment, and user, so it cannot be replayed against a
different target.

**Being authorized and being correct are different questions.** `err-005` says a restart
only applies to a wedged service, and that restarting a *saturated* one makes the
outage worse by removing the capacity still serving. That rule is enforced in the tool
layer, not left to the model: an `sre` who insists on restarting a healthy-but-saturated
service is refused and pointed at scaling out. Guardrails that depend on the model
reasoning correctly are not guardrails.

## What it does

| Capability | Status |
|---|---|
| Retrieval over a help center with citable `doc_id`s | Working (BM25 baseline) |
| Refuses when retrieval is empty, offers `create_ticket` | Working |
| Deprecated docs outranked by their replacement | Working, enforced by invariant |
| Role-gated tools with append-only audit log | Working |
| Human-confirmed restarts, token bound to target | Working |
| Runbook preconditions enforced in the tool layer | Working |
| Golden eval set with per-assertion scoring | Working, 20 cases + 4 held out |
| OpenAI provider | Implemented and unit-tested; **not yet run against the live API** |
| Vertex/Gemini provider | Interface defined, call not wired |
| LangGraph orchestration and session memory | Plain loop today |

Current numbers are in [`docs/EVALS.md`](docs/EVALS.md), regenerated by `make eval`.

**The baseline is deliberately not impressive.** The default provider is `stub`: a
rule-based planner with no model behind it, which exists so the harness runs offline in
CI and so the golden set has a published control. It passes **13 of 20** cases, with
safety at **5 of 6**. That is the bar a real provider has to clear, and the seven
failures are the work queue — they are listed in `docs/EVALS.md` rather than smoothed
away.

Read [the regression log](docs/EVALS.md#regression-log) first if you only read one
thing. Three entries so far, including one finding that a refusal threshold *cannot* be
tuned to separate an unanswerable question from an answerable one on a lexical
retriever, with the measurements that show why.

## How it fits together

```
ticket text
    |
    v
agent loop  --------> provider (stub | openai | vertex)
    |                     returns: tool calls, or a final answer
    v
call_tool  <-- the only entry point to the tool surface
    |  authorize (roles.py) -> confirmation gate -> audit log -> execute
    v
search_docs   lookup_ticket   get_status   get_deploys   create_ticket   restart_service
    |
    v
answer + citations + behavior label  ---> eval harness scores it
```

The one design decision worth calling out: tools are plain Python functions behind a
single `call_tool` dispatcher, and **MCP is a transport in front of that dispatcher**,
not the place the rules live. The agent, the eval suite, and an external MCP client all
traverse the same authorization code. Putting the permission checks in the MCP layer
would have meant the eval suite tested a different code path than production used.

Roles come from the *server* environment, never from the client. A client that could
name its own role would make the permission table decorative.

What the agent must do is in [`docs/PRD.md`](docs/PRD.md), stated as behaviours with the
eval case that owns each one. How it is built is in
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md), with one record per decision in
[`docs/adr/`](docs/adr/README.md) — including the six decisions not yet made. What an
adversary can do to it is in [`docs/THREAT_MODEL.md`](docs/THREAT_MODEL.md).

### Tool permissions

| Tool | `employee` | `sre` |
|---|---|---|
| `search_docs` | yes | yes |
| `get_status`, `get_deploys` | yes | yes |
| `lookup_ticket` | own tickets only | all tickets |
| `create_ticket` | yes | yes |
| `restart_service` | **no** | only after human confirm |

## Evaluation

The eval harness is the part of this repo worth reading first.

Each case declares assertions independently — which documents must be cited, which must
*not* be, which tools are required or forbidden, what the response text must contain,
and which behaviour label is acceptable. A regression therefore names the property that
broke instead of just turning a case red.

Assertions roll up into four reported families (groundedness, correct-tool, behaviour,
content) plus one gate:

**Safety is a gate, not an average.** The permission, unsafe-restart, and unanswerable
categories ship at 100% or they do not ship. A groundedness average of 94% that hides a
permission bypass is worse than useless, so `--require-safety` scores those categories
separately and fails the build on its own.

Case categories: `how_to`, `stale_doc`, `wrong_diagnosis`, `incident_lookup`,
`missing_evidence`, `wrong_service`, `permission_denied`, `unsafe_restart`,
`unanswerable`, `cross_doc`, `escalate`.

Two categories deserve explanation, because they are what keep the numbers honest when
both the corpus and the questions are synthetic:

- **`unanswerable`** asks plausible questions the corpus cannot answer. Without these,
  a groundedness score measures nothing but the retriever's willingness to return
  something.
- **`wrong_diagnosis`** pairs `INSUFFICIENT_CAPACITY` against `QUOTA_EXCEEDED`. They
  look alike and have opposite resolutions — retrying a different zone fixes one and can
  never fix the other. Confusing them is the most likely real-world failure.

`data/evals/holdout.jsonl` is never tuned against. It exists so that a golden-set score
climbing over four weeks can be checked against something that was not used to get
there.

```bash
make eval                                   # golden set, writes docs/EVALS.md
psa-eval --file holdout.jsonl               # held out; do not tune against this
psa-eval --require-safety --fail-under 0.8  # CI gates
```

## Layout

```
data/corpus/        7 synthetic help-center articles (one deliberately deprecated)
data/fixtures/      fake tickets, environments, deploy history
data/evals/         golden.jsonl (20 cases), holdout.jsonl (4)
src/psa/roles.py    the permission table: the security boundary, one file, no deps
src/psa/rag/        chunking, BM25 index, Retriever seam
src/psa/mcp_server/ tools, call_tool dispatcher, audit log, MCP stdio transport
src/psa/agent/      loop, system prompt, behaviour policy
src/psa/providers/  provider interface; stub / openai / vertex
src/psa/evals/      scoring and the runner
ui/app.py           Streamlit support console
```

The core — retrieval, permissions, tools, eval harness — is **stdlib-only** on purpose,
so `make eval` works on a fresh clone with no API key and no network. Model SDKs are
optional extras.

## Roadmap

Honest status: weeks 2 through 4 are not done.

- **Week 2.** Run the OpenAI provider against the live API and record its scores beside
  the stub baseline. Port the loop to LangGraph for session memory and resumable
  confirmation. Close the last safety case (see regression log R2).
- **Week 3.** Vertex/Gemini skin, and swap BM25 for embeddings *behind the existing
  `Retriever` interface* so the golden set measures whether it actually helped.
- **Week 4.** Write up one eval miss end to end — symptom, root cause, fix, score
  movement — in the regression log. Record p95 latency and cost per request.
- **Week 5, if it earns it.** Vertex Vector Search, Cloud Run, an ADK skill wrapper.

## Non-goals

Not a coding agent. Not an SRE incident copilot — incident triage is one ticket type
here, not the product. No real customer data, ever. No fine-tuning: the interesting
problems in this shape of system are retrieval quality, permissions, and evaluation,
and none of them are solved by training a model.