# agent-telemetry-mcp

An MCP server that reports what your AI agents cost: tokens, latency, dollars per
request. Each caller sees only their own rows, because the customer id comes from
their OAuth token rather than from a tool argument.

There is also an ADK agent that asks it questions in English.

MCP Python SDK 2.0 (protocol revision 2026-07-28), ADK 2.6, Auth0, BigQuery, Cloud Run.

## Try it without installing anything

It is deployed. An unauthenticated call tells you where to authenticate:

```console
$ curl -si https://agent-telemetry-x62fjiecda-ew.a.run.app/mcp \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json, text/event-stream' \
    -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="Authentication required",
  resource_metadata="https://agent-telemetry-x62fjiecda-ew.a.run.app/.well-known/oauth-protected-resource/mcp"
```

That last URL is the RFC 9728 document, and it is public — it names the
authorization server and the scopes this resource accepts.

## What it looks like

![two tenants, one binary](docs/demo.gif)

Two logins against the same deployment. `demo@acme.com` sees about $0.29;
`demo@globex.com` sees about $0.49 — a bigger bill from a third as many calls,
because that tenant is seeded pro-heavy. The totals creep up as you use it, since
`CostPlugin` records each run. Then the model is taken out of the loop and the tool
is called directly with a tenant named in the arguments:

```
{'days': 7}                       -> tenant=globex  cost=$0.4916
{'days': 7, 'tenant': 'acme'}     -> tenant=globex  cost=$0.4916
{'days': 7, 'tenant_id': 'acme'}  -> tenant=globex  cost=$0.4916
```

Both accounts are on a throwaway Auth0 dev tenant, and both can write:
`demo@acme.com` / `AcmeDemo-20b9c40a64-Xq7`, `demo@globex.com` / `GlobexDemo-6d9b1808cf-Kt3`.

## Why a server and not just ADK's BigQueryToolset

`BigQueryToolset` accepts an end user's token:

```python
credentials = Credentials(token=user_token)
BigQueryToolset(credentials_config=BigQueryCredentialsConfig(credentials=credentials))
```

That is fine when you trust the agent. It is not enough when one agent serves many
customers, because nothing checks the token — the agent uses whatever it was handed.

So the server does the checking. It validates issuer, audience and scopes, reads
the customer id from the validated token, and binds that id as a query parameter.
ADK's own docs say the same about their nearest equivalent:

> `job_labels`: Note: These labels are for usage discovery and tracking purposes
> only and should not be used for security-sensitive decisions.

## How it fits together

```
telemetry_analyst              picks which sub-agent answers
├── explorer_agent    ──→  BigQueryToolset   public datasets, agent's identity
└── telemetry_agent   ──→  this MCP server   your own data, your identity

CostPlugin  ──→  this MCP server  ──→  BigQuery table
```

`CostPlugin` runs on the ADK `Runner`, so it sees every model call from every
agent in the tree. After each call it reads the token counts, prices them, and
sends a row through the same authenticated connection the reads use. That is
where the data in the table comes from — the system measures itself.

`explorer_agent` is four lines of `BigQueryToolset`. Read-only mode and the byte
ceiling are already `BigQueryToolConfig` fields, so there was nothing to write.

## The one rule

No tool takes a `tenant` argument. Not a validated one — there is no such parameter:

```
usage_summary          days
cost_by_model          days
slowest_invocations    days, limit
record_invocation      agent, model, prompt_tokens, output_tokens, ...
```

`record_invocation` needs `telemetry:write`, the reads need `telemetry:read`, and
a token that lacks a scope does not see the tool it would have unlocked. Asking
anyway returns 403 with an RFC 6750 `scope` hint, which the client uses to step up
in one round trip. `tests/test_tenancy.py` fails the build if a tenant argument
ever appears.

## Run it

```bash
uv venv && uv pip install -e ".[dev,agent]"
cp .env.example .env          # AUTH0_DOMAIN, MCP_CANONICAL_URI, GOOGLE_CLOUD_PROJECT
                              # plus GOOGLE_API_KEY for the model

uvicorn agent_telemetry.server:create_app --factory --reload
python -m agent.main
```

`deploy/deploy.sh` provisions and deploys; the Auth0 steps that have no API are in
[`deploy/README.md`](deploy/README.md). `deploy/seed.py` fills a demo tenant, priced
from `agent/pricing.py` so the demo cannot drift from the code it is demonstrating.

## Tests

```bash
pytest        # 43 tests, no credentials needed
```

| file | what it covers |
|---|---|
| `test_tenancy.py` | two tokens get two different slices; no tenant claim is refused |
| `test_scope_gate.py` | per-tool scopes, through the real ASGI stack |
| `test_agent_auth.py` | the client half: loopback callback, cached tokens, discovery |
| `test_cost_plugin.py` | pricing, and not double-counting streamed responses |
| `test_wiring.py` | the plugin and the server still agree on field names |

## Things that were not obvious

**The tests all passed while the deployed server could not answer a single
authenticated request.** An in-memory `Client(server)` hands the server an object
graph — no HTTP, no Host header, no ASGI receive channel. Four separate faults
lived in that gap. The one I would not have guessed: `streamable_http_app()`
defaults to a localhost-only Host allowlist, so every Cloud Run request came back
421, and only after authentication had succeeded.

**`scopes_supported` and `required_scopes` are different things.** The SDK builds
the RFC 9728 document from `required_scopes`, which is the floor a token must
already clear, not what a client may ask for. So the document advertised
`telemetry:read` alone, and a client that follows the spec's scope-selection order
was told `telemetry:write` did not exist.

**The client discovers metadata on the 401 path only.** A process that starts with
a cached token never sees a 401, so a later 403 step-up re-authorized with no
metadata in hand and fell back to `{resource_server}/authorize` — a URL this server
does not serve. The browser opened on a 404 and the flow waited forever.

**ADK's `McpToolset` does not work with MCP SDK 2.x.** It imports
`mcp.shared.session`, which 2.0 removed, and the package swallows the ImportError
and logs it at DEBUG. What you see is `cannot import name 'McpToolset'`. That is
why `agent/toolset.py` exists.

## Not included

Approval prompts, the Tasks extension, MCP Apps, A2A, ADK evals. All doable, none
of them make the access control any better.