# ABC Vision MCP

An MCP server that exposes a production badminton video-analysis engine as
agent tools — and records, at every branch point, how a measured visual result
changed what the system did next.

Built on **OpenCV 5.0.0**. Speaks **MCP over Streamable HTTP**.

---

## It is running. Go and look.

| | |
|---|---|
| Simulated Alexa+ experience | **https://d2xks2we90iwvv.cloudfront.net/** |
| The MCP server it calls | `https://d2xks2we90iwvv.cloudfront.net/mcp` |

![The demo page: a staged conversation on the left, ten real MCP calls on the right](docs/demo_page.png)

The conversation on the left is staged. Every call on the right is real — this
browser talking to that server over Streamable HTTP, each one openable to its
raw JSON-RPC. The page is served from the same origin as `/mcp`, which is why
there is no proxy in between and nothing that could be a recording.

The page runs the loop twice. First the ordinary case, where the player is on
court for all three rallies: identity locks on the first attempt, nothing is
excluded, and analysis proceeds. Then the harder one, where the player sits out
the third and no assignment of detections to people can cover a rally they were
not in — the loop tries every anchor, reaches 2 of 3, and asks instead of
publishing.

Only the second run branches, and the branch is the point. The first run is
there because the page used to show only the second, which reads as a system
that never answers.

**Verify the OpenCV version yourself** rather than taking this README's word
for it — call `protocol_info` on that endpoint and it reports the version of
the process that answered you.

Opening `/mcp` in a browser returns a JSON-RPC error about a missing session
ID. That is correct: it is a protocol endpoint, not a web page. The page is at
`/`.

---

## Why this exists

ABC (AI Badminton Coach) is a shipping product: an App Store listing, seven
locales, five jurisdictions' privacy compliance, 527 commits, and 1,117
production jobs over the twelve days to 2026-09-17. Its defining behaviour is
that it refuses to answer when it is not sure — of the 705 jobs that produced a
report, it declined to assign a skill level in **42.7%** of them, and recorded a
human-readable reason each time. The largest single reason, too little of the
player tracked, accounts for 30.2% of those same 705.

That refusal is not a UI message bolted on at the end. It is the outcome of a
chain of gates, each fed by a measurement from the vision pipeline: was a court
found, did the camera drift, was the player held long enough, were the serves
segmented cleanly. When a gate fails, the system does something different —
retries with a different anchor, asks the player to identify themselves, or
publishes an analysis with the level withheld.

This server makes that chain callable by an agent, and writes down the
reasoning as it goes.

## What is in here

| Path | What it is |
|---|---|
| `src/abc_vision_mcp/server.py` | The MCP server and its 16 tools |
| `src/abc_vision_mcp/identity.py` | The agent loop: propose, score, exclude the anchor, propose again |
| `src/abc_vision_mcp/trace.py` | Decision trace: tool calls, branch points, and the alternatives not taken |
| `src/abc_vision_mcp/agent_core.py` | Provider-agnostic tool-calling agent: loop, grounding guard, caps, rate limits, cost |
| `src/abc_vision_mcp/agent_providers.py` | OpenAI-compatible and Nebius model adapters, price table |
| `src/abc_vision_mcp/coach_agent.py` | This project's wiring: the tool allowlist, prompt, fallback text |
| `src/abc_vision_mcp/stats.py` | Production statistics, always with both denominators |
| `src/abc_vision_mcp/web/index.html` | The simulated Alexa+ surface served at `/` |
| `src/abc_vision_mcp/demo_fixture.py` | The evidence the demo runs on, shared by the page and the trace script |
| `scripts/demo_agentic_loop.py` | Reproduces the demo run and writes the decision trace |
| `scripts/make_outcomes_chart.py` | Renders `docs/outcomes.svg` from the record — no figure is typed by hand |
| `tests/` | 127 tests, including the redaction properties the trace must hold |
| `data/deidentified/` | Production outcome statistics with every identifier removed |
| `infra/` | AWS deployment: bundler, Dockerfile, CodeBuild |
| `docs/AWS.md` | What runs where, and why it is not what was planned |
| `docs/CORRECTIONS.md` | **What this project published and then found to be wrong** |

## Decisions, not logs

A log of "scan ran, then lock ran" proves nothing — a fixed script produces the
same log. So the trace separates two kinds of record:

- **ToolCall** — what ran, with inputs and outputs. Supporting detail.
- **Decision** — a branch point where a *measured value* selected the next
  action, recorded together with the branches that were **not** taken and the
  rule that maps evidence to branch.

`summary()` reports `branching_decisions` separately from `decisions`, because
a decision with no alternatives did not branch on anything and should not be
presented as if it had.

Example of one real branch, taken from the engine's identity lock
(`core/analysis.py`, `MAX_ANCHOR_ATTEMPTS = 8`): when geometric validation of a
candidate anchor fails, the rejected anchor key is added to an `avoid` set and
the *same tool is called again with different parameters*. The visual result
changed the next tool call. That loop already exists in production; this server
exposes and records it rather than inventing one for a submission.

## Privacy

This repository is built to be handed to competition judges, so it is
structured to make leaking hard rather than to rely on care:

- `.gitignore` is an **allowlist** — everything is ignored, and tracked paths
  are named explicitly. A sibling repository used a blocklist, missed several
  directories, and ended up tracking customer data it should never have held.
  This one starts from the opposite default.
- The trace redacts by key name, so a field added later called `subject_name`
  is redacted the day it appears. Paths and identifiers become truncated
  SHA-256 handles: stable enough to show two calls hit the same video, useless
  for recovering which video.
- Only de-identified aggregates ship in `data/`. No video, no frames, no
  subject records.
- Redaction is covered by tests, not by convention.

## Requirements

- Python 3.12
- `mcp` 2.2.0 — **negotiated protocol version `2025-11-25`**
- OpenCV 5.0.0 (`opencv-python-headless==5.0.0.93`)

> The SDK also exposes `LATEST_PROTOCOL_VERSION = 2026-07-28`. That is the
> transport revision, not the protocol version, and it is always the higher of
> the two — which makes it the tempting one to quote. An earlier version of
> this README quoted it. See `docs/CORRECTIONS.md`.

## Running

```bash
python -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt

# stdio, for a local client
.venv/Scripts/python -m abc_vision_mcp.server

# Streamable HTTP on http://127.0.0.1:8931/mcp, with the demo page at /
.venv/Scripts/python -m abc_vision_mcp.server --http
```

Serving on a public interface? Name the hosts you answer to. The SDK enables
DNS rebinding protection only for a loopback bind, so a container that binds
`0.0.0.0` has none unless it asks:

```bash
.venv/Scripts/python -m abc_vision_mcp.server --http --host 0.0.0.0     --allowed-host your.public.name --allowed-origin https://your.public.name
```

Without it the server starts anyway and prints a warning saying what is not
protecting it. `protocol_info` reports the resulting state, so a deployment can
be checked rather than believed.

Tests:

```bash
.venv/Scripts/python -m pytest tests/ -q
# 115 passed, 12 skipped   without an engine checkout
# 127 passed              with ABC_ENGINE_PATH set
```

The twelve skips are the tests that need the analysis engine, which is not in
this repository and never will be — see *Responsible operation* above. Point
`ABC_ENGINE_PATH` at a checkout containing `core/identity/lock.py` to run them.
Nothing else in the suite depends on it, so a reviewer without the engine still
exercises the MCP surface, the transport security, the redaction and the demo
fixtures.

Reproduce the decision trace shown on the demo page — no video, no customer
data, production tools:

```bash
.venv/Scripts/python scripts/demo_agentic_loop.py --out traces
# 6 tool calls, 4 decisions, 4 branching, 1 human approval request
```

Check a running server against the spec, with the checker we published for this
(MIT, separate repository):

```bash
pip install git+https://github.com/ekyo0224/mcp-conformance
mcp-conformance https://d2xks2we90iwvv.cloudfront.net/mcp   # 12 checks, 0 failures
```

## Status

Deployed and public, running on AWS for the judging period of two 2026
competitions. 16 tools, 127 tests, OpenCV 5.0.0, negotiated protocol
`2025-11-25`.

The analysis engine itself is **not** in this repository and never will be. A
named allowlist of eleven files is staged at build time inside the owner's own
AWS account, and the build refuses to run if anything resembling customer data
reaches it.

## Nemotron coach notes (NVIDIA × Nebius)

`explain_report` (and the "Coach notes" block on the demo page, whose model
version is behind a "Write with Nemotron" button) turns a report's summary into one to three plain coaching suggestions and, when
a level was withheld, an honest sentence on why. The wording comes from an
NVIDIA Nemotron model served by Nebius Token Factory; the facts do not.

```bash
export NEBIUS_API_KEY=...          # never logged, never returned
export NEBIUS_MODEL=...            # optional; default nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B
export NEBIUS_BASE_URL=...         # optional; default https://api.tokenfactory.nebius.com/v1
```

- **What is sent:** only `status`, `level` (if the report had one),
  `rallies_total`, `rallies_covered` and `decline_reason` (only if a tool supplied one). No video, frames,
  names or paths; other fields are dropped before the call. What the summary
  fields themselves reveal is the rally counts and the published/declined status.
- **Fallback:** with no key, a timeout (8 s), an HTTP error, or an answer that
  states a level the report lacks, uses a number the report lacks, or breaks the
  length limits, the result is fixed rule-based text from the same fields. Every
  result carries `source: "nemotron"` or `source: "rules"`.
- **Not claimed:** the model never decides or produces a level. The model path
  has been tested here against mocked responses; nothing in the test suite
  calls Nebius.

## Ask the coach: a Nemotron agent that calls our tools

`ask_coach` (MCP tool), `POST /demo/ask` (`{"question": "..."}`, 300 characters
max) and the "Ask the coach (Nemotron agent)" box on the demo page. A player or
coach asks in plain language; an NVIDIA Nemotron model on Nebius Token Factory
decides which tools to call (OpenAI-style tool calling), the tools run
in-process, and the model answers from their results. The result is
`{answer, tools_called, source, model, usage}`; `source` is `nemotron` or
`rules`, and `rules` carries a `reason` (`no_key`, `cap`, `ungrounded`,
`max_rounds`, `error`, `budget`).

```bash
export NEBIUS_API_KEY=...            # never logged or returned
export NEBIUS_AGENT_MODEL=...        # optional; default nvidia/nemotron-3-super-120b-a12b
export NEBIUS_DAILY_CALL_CAP=200     # optional; model calls per UTC day
# other OpenAI-compatible host: AGENT_PROVIDER=openai-compatible with
# AGENT_API_KEY, AGENT_BASE_URL, AGENT_MODEL
.venv/Scripts/python -m abc_vision_mcp.server --http
```

**Tool allowlist** (`coach_agent.py`, read-only, in-process on bundled data,
no network, no video): `production_outcomes`, `decline_reasons`,
`human_approval_dropoff`, `demo_case_report` (the demo cases' summaries, the
same function behind the page's coach notes). A request for any other tool name
is refused and the model is told so. Nothing that uploads, writes or needs a
real video is reachable.

**Guards, all in code and covered by tests:**

- *Grounding.* Every number in the final answer must appear in the tool results
  or the question; a skill grade is only allowed if a tool returned one. A
  failing answer gets one correction and one more try; then the answer is
  replaced by fixed text built from the tool results (`source: "rules"`).
  Spelled-out quantities (one to twenty, "twice", "half") are checked against
  the same sources.

  **What this does not check:** wording-level claims. An answer can use only
  numbers the tools returned and still say something the tools did not (a
  cause, a rule, a comparison), be vague, or answer a neighbouring question,
  and the guard will pass it. Tool descriptions are kept to what each result
  actually returns, and the prompt forbids explaining beyond it, but that is
  an instruction, not a check.
- *Limits.* At most 4 rounds of tool calls, 600 output tokens and 20 s per
  model call, 75 s per question. HTTP errors and timeouts fall back at once and
  are never retried. No threads, timers or loops beyond the round cap.
- *Cost cap.* A process-wide daily cap on model calls (default 200), counting
  `ask_coach`, `/demo/ask`, and the model path of `explain_report` and
  `POST /demo/explain`; past it the rules answer is returned with reason `cap`,
  with no call made.
- *Per client.* Web visitors: 5 requests a minute and 30 a day, one in flight at
  a time. MCP callers share one bucket: 100 a day, 2 in flight. Over the limit
  is HTTP 429 (a tool error for the MCP tools) and Nebius is not called.
  Identical questions are served from a 10-minute cache. The client key is the
  `CloudFront-Viewer-Address` header (port dropped) when present, else the
  socket address; `X-Forwarded-For` is ignored because a caller can set it. The
  table of client keys drops day-old entries and is capped at 10,000.
- *Nothing spends on page load.* The demo page's coach notes load as fixed rules
  text (`GET /demo/explain` never calls a model). The model versions
  (`POST /demo/explain`, `/demo/ask`) are sent only from a click or submit.
- *What the model sees.* The question, the allowlisted tools' names and
  descriptions, and their results: published outcome statistics, decline-reason
  categories in plain words (for example "the camera moved too much"), the
  demo cases' rally counts, and the long-upload figures. No video, frames,
  names, paths or source code. The decline categories do name stages of the
  analysis in plain language, and the results carry the project's own outcome
  codes; they carry no thresholds, margins or scoring values.

**Model.** Default is `nvidia/nemotron-3-super-120b-a12b`; the Nano model
(`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`) also calls tools correctly and is
faster and cheaper, so it is a reasonable `NEBIUS_AGENT_MODEL`. Thinking is
always switched off (`chat_template_kwargs.enable_thinking=false`).

**Cost.** `usage` sums the API's token counts over a question and prices them
from a table in `agent_providers.py` (Super $0.30 / $0.90 per 1M prompt /
completion tokens, read from the account's `/models?verbose=true`); a model
missing from the table reports `usd: null` rather than a guess. Measured live (6 questions, Super):
about 3,100 prompt and 220 completion tokens, roughly $0.0011, per question.

**Reuse.** `agent_core.py` imports nothing from this server: bring a
`Provider`, register your own tools in a `ToolRegistry`, and pass an
`AgentConfig` (system prompt, fallback, extra checks).

**Not claimed.** The agent cannot analyse a video and never produces a skill
level. The tests mock the network; the live runs were small manual checks.