Triage Desk

An internal Trust & Safety / CX triage tool for a live-shopping marketplace. A report comes in; the system retrieves the governing policy, classifies the report against it, and decides whether the recommended action is safe to take without a human.

Built as a work sample for Whatnot's AI Engineer role. It is small on purpose — the interesting part is not the classifier, it's everything wrapped around the classifier that makes an off-the-shelf model dependable enough to sit in front of an ops queue.

report ──▶ retrieve (BM25 over policy corpus)
             │
             ▼
        classify (Claude, structured output + prompt caching)
             │
             ▼
        guardrails ──▶ verdict + cited policy + auto-actionable?
             │
             ▼
        telemetry (latency / tokens / cost / guardrail rate)

Run it

bun install
bun start          # http://localhost:3020
bun test           # 20 unit tests
bun evals/run.ts   # eval + regression gate
bun mcp/server.ts  # same pipeline over MCP (stdio)

Runs with no API key by default: TRIAGE_MODE=mock uses a deterministic keyword baseline that produces the identical schema. Set TRIAGE_MODE=live (plus ANTHROPIC_API_KEY, or an ant auth login profile) to route classification through Claude.

The mock arm is not a stub — it's the control. It keeps CI free and flake-free, and it's the baseline the model has to beat before "use an LLM here" is a justified decision.

Related MCP server: mcp-incident-copilot

What's actually in here

Retrieval (src/retrieval.ts) — BM25 over the policy corpus, written directly so there's no index to stand up. Swap it for pgvector or embeddings and nothing above it changes.

Classification (src/llm.ts) — Claude with a JSON Schema via output_config.format, so the response is guaranteed to parse; no regex-extraction or retry-on-parse loop. The system prompt is a stable prefix carrying cache_control, so repeat traffic reads the cache instead of re-billing the instructions.

Guardrails (src/triage.ts) — the model proposes; these rules decide what may happen without a human:

Evals (evals/) — 14 labeled reports, graded on three axes: category accuracy, retrieval recall@3, and escalation recall. Escalation is gated at 1.00 and cannot be tuned down: a run that classifies well but lets one minor-safety report auto-resolve is a failing run. Category accuracy is graded against the arm being run, because holding a keyword baseline to the model's bar isn't a meaningful test.

Telemetry (src/telemetry.ts) — p50/p95 latency, token counts (including cache reads), cost at list price, guardrail firing rate, and the category/action distribution. Exposed at /api/ops and as an MCP tool.

MCP server (mcp/server.ts) — triage_report, search_policy, and triage_ops. Any MCP client gets policy-grounded triage without reimplementing retrieval, prompting, or the guardrails. This is the reusable half: the web UI and an agent call the same pipeline.

Three bugs found while building this

The first two surfaced on the first eval run, before any of this had been near a real report. The third never showed up in a metric at all — every number was green — and was only visible in the rendered output. They're written up rather than quietly fixed because they're the argument for having the harness at all.

1. A retrieval miss from missing stemming. A report about "vape pens" never matched the policy line about "vaping products" — different tokens, zero BM25 overlap, governing policy silently absent from the context. Fixed with conservative suffix stripping (src/retrieval.ts), covered by a test.

2. Escalation depended on the classifier being right. A report about a 15-year-old streamer was classified no_violation, so no category guardrail fired and it would have auto-resolved. Category-based rules can't be the only path to a human when the classifier is the thing that failed. Fixed with the deterministic safety screen, which runs on the raw report and overrides the verdict regardless of category or confidence.

The second one is the reason escalation recall is a separate graded axis instead of being folded into accuracy.

3. Citations that the verdict never relied on. Visible only in the UI, not in any metric: the classifier was citing the top-2 retrieved policies rather than the ones matching its own verdict, so a report about Zelle payments came back tagged off_platform_transaction while citing the harassment policy. Every number in the eval was green. Fixed by citing only retrieved policies whose category matches the verdict.

That one is worth its own note: an ungrounded citation is the failure the no_policy_cited guardrail exists to catch, and the system was manufacturing it internally. Rendering the intermediate state — what was retrieved, in what order, versus what was cited — is what surfaced it. The RETRIEVED line in the UI still shows the harassment policy out-ranking the correct one on that query, which is honest: BM25 precision on a 10-document corpus is mediocre, recall@3 is what the pipeline actually depends on, and that's the metric the eval grades.

Notes on what I did not do

• No fine-tuning or model training — off-the-shelf model, engineering around it.

• No vector DB. At 10 policies BM25 wins on latency and operability; the interface is the part worth getting right.

• The policy corpus is written for this prototype. It's realistic in shape (indicator lists, severity tiers, escalation carve-outs) but it isn't anyone's real policy.

Neal Rodriguez · notifyneal@gmail.com · port.ragflo.com