Skip to main content
Glama
io.github.speedofred/ami-survey

io.github.speedofred/ami-survey

by speedofred

AMI survey client

Measures what a completed agent workflow cost to run, by reading the runtime's own session log, and submits the result to the AMI survey at survey.agentbenchmark.dev.

The cost of a single API call is easy to obtain. The cost of one finished piece of work is not: one triaged ticket, one screened CV, one drafted reply, across every call, retry and tool round-trip the agent made getting there. This measures that figure.

Every number is read from the runtime's records rather than reported by the agent. An agent asked how many tokens it has just used will estimate, and will present the estimate with confidence.

What comes back

A real run of six support tickets, triaged and answered by Claude Opus 5 in Claude Code:

Maturity Index      85.0  Strong          (observability 40%, evidence 30%, quality 30%)
Performance         78.13 Strong          confidence Very High

  quality           80.0   graded Good on ami-quality-v2
  cost              72.73  $0.123226 per ticket   ($0.739355 for the run)
  speed             84.47  20.90s per ticket
  evidence          70.0   measured, on a self-issued token
  observability    100.0

findings
  weakness  Cost is the weakest pillar at 72.73; speed is strongest at 84.47.
            $0.123226 per unit against a $0.01 reference. A cheaper model, or
            fewer calls, moves this; check calls[] for where the tokens went.

  note      Cost and speed were scored against a provisional reference, which is
            a placeholder rather than a measurement. Do not quote them as settled
            yet. The Maturity Index does not use the reference and is unaffected.

$0.12 per ticket, 21 seconds per ticket. That is the figure this exists to produce, and it is the one most teams cannot currently state about their own work.

The findings record where the scorecard's own numbers are soft. A cost reference that is still a placeholder is reported as a placeholder rather than folded into the score.

Related MCP server: ClaudeCode Session Notes MCP Server

Install

There are two routes. What separates them, and what separates the hosts below, is whether anything can read your runtime's logs. That determines whether the result is recorded as measured or unmeasured.

Install the client

One block of configuration, then restart the agent.

Claude Code, in ~/.claude.json, or through the Developer settings in Claude Desktop:

{ "mcpServers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }

Runs are measured: the client reads Claude Code's own session log, so the token counts and the cost come from the runtime rather than from the agent.

In ~/.codex/config.toml. Codex uses TOML here, not JSON:

[mcp_servers.ami-survey]
command = "uvx"
args = ["ami-survey"]

The client reads Codex's rollout log. This also covers a local or self-hosted model driven through Codex, because the log is written either way.

In .vscode/mcp.json, or your user profile through the MCP: Open User Configuration command. The key is servers here, not mcpServers:

{ "servers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }

The tools work and the survey submits, but Copilot writes no session log the client can read. The run carries the workflow, the grade and the stage timings without token counts or cost, and is recorded as unmeasured.

Then, once the agent finishes a piece of work, ask it:

Take the AMI survey regarding the ticket triage you just did

There is nothing to clone and nothing to keep updated. No token needs to be supplied: the first call that requires one registers the machine and stores the token at ~/.ami-survey/token.

uvx is part of uv, the tool the MCP documentation uses for Python servers. If uv is not installed, GETTING-STARTED.md gives a pipx form and a route that requires neither.

If you already hold a token, set it as AMI_API_TOKEN in that block's env and it will be used instead of registering a new one.

Or use the remote connector, with nothing installed

In claude.ai, open Settings, then Connectors, then Add custom connector, and supply:

https://survey.agentbenchmark.dev/mcp

Nothing is installed and no token is required. These runs are recorded as unmeasured and are never compared against measured ones: a remote server cannot read your runtime's logs, so token counts and cost are absent rather than estimated.

Licence

MIT. Use it, modify it, redistribute it, include it in your own work.

The licence covers this client only. It grants no rights in the survey service the client submits to, or in the data that service collects.

Full terms in LICENSE.

What leaves your computer

Token counts, timings, model names, the stage names the workflow declared, and the grade. Not your files, not your prompts, not your shell commands. GETTING-STARTED.md sets this out in full.

Submissions go to survey.agentbenchmark.dev and nowhere else. That destination is a constant in the source rather than a setting, so a stale environment variable cannot redirect a submission onto your own disk. That is the one failure which would make a run appear successful while collecting nothing.

Requirements

Python 3.9 or newer. No dependencies; the standard library only.

Further reading

  • GETTING-STARTED.md assumes no prior setup, covers macOS, Linux and Windows, and explains what to do when the tools do not appear.

  • COMMANDS.md documents the clone-and-run route, which benchmarks one workflow across several models on your own API key. It is not required in order to take part.

  • MAKE-IT-MEASURABLE.md explains how to structure a workflow so that there is something worth measuring.

Available Tools

14 tools
ami_collect_telemetryA

Measure the workflow run: detects which agent runtime you are in, reads that runtime's own call records (token usage, model, timings reported by the provider), attributes each call to a stage or observed execution phase, and stores the result on the run. Returns the collected inventory fields. Use these numbers verbatim. If your runtime has no adapter, this says so and you should use ami_record_calls instead.

Inputs: cwd, run_id, adapter, session_id, +3 more.

ami_get_grading_scaleA

Return the AMI output-quality grading scale. Read this before grading: agent_output_grade must be one of its codes and is validated on submit.

Inputs: no named parameters.

ami_get_instructionsA

Return the AMI survey procedure as markdown. Call this FIRST if your harness has no skill mechanism that already gave you the procedure - it tells you the call order and, for your runtime, how telemetry is obtained. Claude Code agents already have it as a skill and do not need this.

Inputs: runtime.

ami_get_reportA

Render the human-readable Markdown report for a run (defaults to the active run). With no run and no active run, returns the index of all submitted surveys.

Inputs: run_id.

ami_get_scorecardA

The scorecard for a submitted run: the AMI Maturity Index, the Performance Score, the five pillars, and structured findings. Every number and finding is computed from the run's own data - the server calls no model. If a human wants this read back as prose, write it yourself from narration_brief.findings, following the instructions there.

Inputs: run_id.

ami_get_surveyA

Return the survey definition: every field from Collection_Inventory.csv, how each one is obtained, and which ones you must answer yourself.

Inputs: no named parameters.

ami_get_workflow_categoriesA

The workflow categories a run may declare itself into, and what each one covers. A category decides which other workflows this run is compared against, so read the list rather than inventing a label.

Inputs: no named parameters.

ami_list_surveysA

List all submitted survey responses and where they are stored.

Inputs: no named parameters.

ami_mark_stageA

Declare the workflow stage you are entering, e.g. 'Classify Severity' or 'Draft Customer Reply'. Call it as you move through the workflow to get a declared-stage Agent Effort Profile; without markers the profile falls back to AMI-observed execution phases. No survey run is needed first: markers emitted before ami_survey_begin are buffered with the timestamp you emitted them at and attached to the run when it opens. When the last stage is done, call once more with closes=true - each marker ends the stage before it, so without a closing one the final stage runs to the end of the measurement window and absorbs everything you do afterwards.

Inputs: cwd, note, stage, closes, +2 more.

ami_record_callsA

Telemetry entry point for agents NOT running in Claude Code. Post the provider-reported usage for each API call the workflow made. Every record needs model, start_time, end_time, input_tokens, output_tokens - values read from real API responses, never estimated.

Inputs: calls (required), run_id, adapter, replace, +1 more.

ami_submit_surveyA

Submit and persist the survey. Requires a grade from the AMI grading scale, a justification, and evidence (the concrete artifacts produced). Writes JSON + Markdown + a CSV index row to disk and returns the paths.

Inputs: grade_evidence (required), agent_output_grade (required), grade_justification (required), grader, +5 more.

ami_survey_beginA

Open an AMI survey for the workflow you have just completed. Detects the runtime, session and measurement window automatically, and adopts any stage markers you buffered with ami_mark_stage while working. Call this FIRST, after the workflow's real work is finished, so the survey's own token spend is excluded from the measurements.

Inputs: workflow_name (required), workflow_description (required), cwd, adapter, +7 more.

ami_survey_statusA

Show the current values collected for the active run, which inventory fields are still empty, and what is blocking submission.

Inputs: run_id.

ami_write_findingsA

Write the judgement sections of a run's scorecard. The server computes every number and the three sections that follow from them; these four are readings of the work that no arithmetic produces, so they are yours to write. Call ami_get_scorecard first and use narration_brief.sections_awaiting_you - it carries the brief for each. Ground every sentence in the run's own evidence; do not invent industry context you were not given.

Inputs: run_id (required), key_finding, industry_next_step, workflow_next_step, +2 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 14 tools

    Disambiguation4/5

    Tools are mostly distinct: telemetry has two entry points (ami_collect_telemetry vs ami_record_calls) that could be confused, but descriptions clearly separate by runtime. Other tools target distinct actions (begin, mark, submit, status, etc.).

    Naming Consistency4/5

    All tools follow the 'ami_' prefix plus clear verb_noun pattern (e.g., ami_get_scorecard, ami_write_findings, ami_submit_survey). Minor deviation: ami_survey_begin uses noun_verb instead of strict verb_noun, but overall consistent.

    Tool Count5/5

    14 tools is well within the sweet spot for a domain as complex as survey instrumentation and reporting. Each tool serves a distinct lifecycle step (setup, telemetry, grading, submission, reporting), and none feel redundant.

    Completeness4/5

    The tool surface covers the full survey lifecycle: begin, collect telemetry, mark stages, get instructions/scale/survey, check status, write findings, submit, and retrieve reports/lists. Minor gap: no explicit tool for updating or amending a run after submission, but that is not core to the stated purpose.

    Maintenance

    ActivityStale
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers