io.github.speedofred/ami-survey
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@io.github.speedofred/ami-survey Take the AMI survey regarding the ticket triage you just did"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
AMI survey client
Measures what a completed agent workflow cost to run, by reading the runtime's
own session log, and submits the result to the AMI survey at
survey.agentbenchmark.dev.
The cost of a single API call is easy to obtain. The cost of one finished piece of work is not: one triaged ticket, one screened CV, one drafted reply, across every call, retry and tool round-trip the agent made getting there. This measures that figure.
Every number is read from the runtime's records rather than reported by the agent. An agent asked how many tokens it has just used will estimate, and will present the estimate with confidence.
What comes back
A real run of six support tickets, triaged and answered by Claude Opus 5 in Claude Code:
Maturity Index 85.0 Strong (observability 40%, evidence 30%, quality 30%)
Performance 78.13 Strong confidence Very High
quality 80.0 graded Good on ami-quality-v2
cost 72.73 $0.123226 per ticket ($0.739355 for the run)
speed 84.47 20.90s per ticket
evidence 70.0 measured, on a self-issued token
observability 100.0
findings
weakness Cost is the weakest pillar at 72.73; speed is strongest at 84.47.
$0.123226 per unit against a $0.01 reference. A cheaper model, or
fewer calls, moves this; check calls[] for where the tokens went.
note Cost and speed were scored against a provisional reference, which is
a placeholder rather than a measurement. Do not quote them as settled
yet. The Maturity Index does not use the reference and is unaffected.$0.12 per ticket, 21 seconds per ticket. That is the figure this exists to produce, and it is the one most teams cannot currently state about their own work.
The findings record where the scorecard's own numbers are soft. A cost reference that is still a placeholder is reported as a placeholder rather than folded into the score.
Related MCP server: ClaudeCode Session Notes MCP Server
Install
There are two routes. What separates them, and what separates the hosts below,
is whether anything can read your runtime's logs. That determines whether the
result is recorded as measured or unmeasured.
Install the client
One block of configuration, then restart the agent.
Claude Code, in ~/.claude.json, or through the Developer settings in
Claude Desktop:
{ "mcpServers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }Runs are measured: the client reads Claude Code's own session log, so the token counts and the cost come from the runtime rather than from the agent.
In ~/.codex/config.toml. Codex uses TOML here, not JSON:
[mcp_servers.ami-survey]
command = "uvx"
args = ["ami-survey"]The client reads Codex's rollout log. This also covers a local or self-hosted model driven through Codex, because the log is written either way.
In .vscode/mcp.json, or your user profile through the MCP: Open User
Configuration command. The key is servers here, not mcpServers:
{ "servers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }The tools work and the survey submits, but Copilot writes no session log the
client can read. The run carries the workflow, the grade and the stage timings
without token counts or cost, and is recorded as unmeasured.
Then, once the agent finishes a piece of work, ask it:
Take the AMI survey regarding the ticket triage you just did
There is nothing to clone and nothing to keep updated. No token needs to be
supplied: the first call that requires one registers the machine and stores the
token at ~/.ami-survey/token.
uvx is part of uv, the tool the MCP
documentation uses for Python servers. If uv is not installed,
GETTING-STARTED.md
gives a pipx form and a route that requires neither.
If you already hold a token, set it as AMI_API_TOKEN in that block's env and
it will be used instead of registering a new one.
Or use the remote connector, with nothing installed
In claude.ai, open Settings, then Connectors, then Add custom connector, and supply:
https://survey.agentbenchmark.dev/mcpNothing is installed and no token is required. These runs are recorded as
unmeasured and are never compared against measured ones: a remote server
cannot read your runtime's logs, so token counts and cost are absent rather than
estimated.
Licence
MIT. Use it, modify it, redistribute it, include it in your own work.
The licence covers this client only. It grants no rights in the survey service the client submits to, or in the data that service collects.
Full terms in LICENSE.
What leaves your computer
Token counts, timings, model names, the stage names the workflow declared, and the grade. Not your files, not your prompts, not your shell commands. GETTING-STARTED.md sets this out in full.
Submissions go to survey.agentbenchmark.dev and nowhere else. That destination
is a constant in the source rather than a setting, so a stale environment
variable cannot redirect a submission onto your own disk. That is the one
failure which would make a run appear successful while collecting nothing.
Requirements
Python 3.9 or newer. No dependencies; the standard library only.
Further reading
GETTING-STARTED.md assumes no prior setup, covers macOS, Linux and Windows, and explains what to do when the tools do not appear.
COMMANDS.md documents the clone-and-run route, which benchmarks one workflow across several models on your own API key. It is not required in order to take part.
MAKE-IT-MEASURABLE.md explains how to structure a workflow so that there is something worth measuring.
Available Tools
14 toolsami_collect_telemetryA
Measure the workflow run: detects which agent runtime you are in, reads that runtime's own call records (token usage, model, timings reported by the provider), attributes each call to a stage or observed execution phase, and stores the result on the run. Returns the collected inventory fields. Use these numbers verbatim. If your runtime has no adapter, this says so and you should use ami_record_calls instead.
Inputs: cwd, run_id, adapter, session_id, +3 more.
ami_get_grading_scaleA
Return the AMI output-quality grading scale. Read this before grading: agent_output_grade must be one of its codes and is validated on submit.
Inputs: no named parameters.
ami_get_instructionsA
Return the AMI survey procedure as markdown. Call this FIRST if your harness has no skill mechanism that already gave you the procedure - it tells you the call order and, for your runtime, how telemetry is obtained. Claude Code agents already have it as a skill and do not need this.
Inputs: runtime.
ami_get_reportA
Render the human-readable Markdown report for a run (defaults to the active run). With no run and no active run, returns the index of all submitted surveys.
Inputs: run_id.
ami_get_scorecardA
The scorecard for a submitted run: the AMI Maturity Index, the Performance Score, the five pillars, and structured findings. Every number and finding is computed from the run's own data - the server calls no model. If a human wants this read back as prose, write it yourself from narration_brief.findings, following the instructions there.
Inputs: run_id.
ami_get_surveyA
Return the survey definition: every field from Collection_Inventory.csv, how each one is obtained, and which ones you must answer yourself.
Inputs: no named parameters.
ami_get_workflow_categoriesA
The workflow categories a run may declare itself into, and what each one covers. A category decides which other workflows this run is compared against, so read the list rather than inventing a label.
Inputs: no named parameters.
ami_list_surveysA
List all submitted survey responses and where they are stored.
Inputs: no named parameters.
ami_mark_stageA
Declare the workflow stage you are entering, e.g. 'Classify Severity' or 'Draft Customer Reply'. Call it as you move through the workflow to get a declared-stage Agent Effort Profile; without markers the profile falls back to AMI-observed execution phases. No survey run is needed first: markers emitted before ami_survey_begin are buffered with the timestamp you emitted them at and attached to the run when it opens. When the last stage is done, call once more with closes=true - each marker ends the stage before it, so without a closing one the final stage runs to the end of the measurement window and absorbs everything you do afterwards.
Inputs: cwd, note, stage, closes, +2 more.
ami_record_callsA
Telemetry entry point for agents NOT running in Claude Code. Post the provider-reported usage for each API call the workflow made. Every record needs model, start_time, end_time, input_tokens, output_tokens - values read from real API responses, never estimated.
Inputs: calls (required), run_id, adapter, replace, +1 more.
ami_submit_surveyA
Submit and persist the survey. Requires a grade from the AMI grading scale, a justification, and evidence (the concrete artifacts produced). Writes JSON + Markdown + a CSV index row to disk and returns the paths.
Inputs: grade_evidence (required), agent_output_grade (required), grade_justification (required), grader, +5 more.
ami_survey_beginA
Open an AMI survey for the workflow you have just completed. Detects the runtime, session and measurement window automatically, and adopts any stage markers you buffered with ami_mark_stage while working. Call this FIRST, after the workflow's real work is finished, so the survey's own token spend is excluded from the measurements.
Inputs: workflow_name (required), workflow_description (required), cwd, adapter, +7 more.
ami_survey_statusA
Show the current values collected for the active run, which inventory fields are still empty, and what is blocking submission.
Inputs: run_id.
ami_write_findingsA
Write the judgement sections of a run's scorecard. The server computes every number and the three sections that follow from them; these four are readings of the work that no arithmetic produces, so they are yours to write. Call ami_get_scorecard first and use narration_brief.sections_awaiting_you - it carries the brief for each. Ground every sentence in the run's own evidence; do not invent industry context you were not given.
Inputs: run_id (required), key_finding, industry_next_step, workflow_next_step, +2 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 14 tools
Tools are mostly distinct: telemetry has two entry points (ami_collect_telemetry vs ami_record_calls) that could be confused, but descriptions clearly separate by runtime. Other tools target distinct actions (begin, mark, submit, status, etc.).
All tools follow the 'ami_' prefix plus clear verb_noun pattern (e.g., ami_get_scorecard, ami_write_findings, ami_submit_survey). Minor deviation: ami_survey_begin uses noun_verb instead of strict verb_noun, but overall consistent.
14 tools is well within the sweet spot for a domain as complex as survey instrumentation and reporting. Each tool serves a distinct lifecycle step (setup, telemetry, grading, submission, reporting), and none feel redundant.
The tool surface covers the full survey lifecycle: begin, collect telemetry, mark stages, get instructions/scale/survey, check status, write findings, submit, and retrieve reports/lists. Minor gap: no explicit tool for updating or amending a run after submission, but that is not core to the stated purpose.
Maintenance
Related MCP Connectors
Cost per run, tool, model and day for AI agents, from one call or an OpenTelemetry export.
Agentic workflow budget approvals with usage receipts.
AI agent cost control: true cost per outcome, budgets, spend limits, anomalies, audit evidence.
Time tracking and invoicing for AI agents and their humans: track, log and bill work by agent.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to self-audit their token usage and costs, providing burn reports, forensics, and optimization recommendations via MCP.104 PyPI113MIT
- AlicenseNot gradedqualityFmaintenanceEnables comprehensive collection and analysis of ClaudeCode development sessions, including agent tracking, tool usage analytics, and performance metrics.MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to track LLM costs, enforce budgets, compare models, and estimate expenses through simple tool calls.-
- AlicenseAqualityBmaintenanceProvides cost analysis of AI agent usage by previewing scrubbed metadata before upload and analyzing spend, cache-hit rates, and waste recovery through a local or remote server.2MIT