Skip to main content
Glama
speedofred

io.github.speedofred/ami-survey

by speedofred

AMI survey client

Measures what a completed agent workflow cost to run, by reading the runtime's own session log, and submits the result to the AMI survey at survey.agentbenchmark.dev.

The cost of a single API call is easy to obtain. The cost of one finished piece of work is not: one triaged ticket, one screened CV, one drafted reply, across every call, retry and tool round-trip the agent made getting there. This measures that figure.

Every number is read from the runtime's records rather than reported by the agent. An agent asked how many tokens it has just used will estimate, and will present the estimate with confidence.

What comes back

A real run of six support tickets, triaged and answered by Claude Opus 5 in Claude Code:

Maturity Index      85.0  Strong          (observability 40%, evidence 30%, quality 30%)
Performance         78.13 Strong          confidence Very High

  quality           80.0   graded Good on ami-quality-v2
  cost              72.73  $0.123226 per ticket   ($0.739355 for the run)
  speed             84.47  20.90s per ticket
  evidence          70.0   measured, on a self-issued token
  observability    100.0

findings
  weakness  Cost is the weakest pillar at 72.73; speed is strongest at 84.47.
            $0.123226 per unit against a $0.01 reference. A cheaper model, or
            fewer calls, moves this; check calls[] for where the tokens went.

  note      Cost and speed were scored against a provisional reference, which is
            a placeholder rather than a measurement. Do not quote them as settled
            yet. The Maturity Index does not use the reference and is unaffected.

$0.12 per ticket, 21 seconds per ticket. That is the figure this exists to produce, and it is the one most teams cannot currently state about their own work.

The findings record where the scorecard's own numbers are soft. A cost reference that is still a placeholder is reported as a placeholder rather than folded into the score.

Related MCP server: ClaudeCode Session Notes MCP Server

Install

There are two routes. What separates them, and what separates the hosts below, is whether anything can read your runtime's logs. That determines whether the result is recorded as measured or unmeasured.

Install the client

One block of configuration, then restart the agent.

Claude Code, in ~/.claude.json, or through the Developer settings in Claude Desktop:

{ "mcpServers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }

Runs are measured: the client reads Claude Code's own session log, so the token counts and the cost come from the runtime rather than from the agent.

In ~/.codex/config.toml. Codex uses TOML here, not JSON:

[mcp_servers.ami-survey]
command = "uvx"
args = ["ami-survey"]

The client reads Codex's rollout log. This also covers a local or self-hosted model driven through Codex, because the log is written either way.

In .vscode/mcp.json, or your user profile through the MCP: Open User Configuration command. The key is servers here, not mcpServers:

{ "servers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }

The tools work and the survey submits, but Copilot writes no session log the client can read. The run carries the workflow, the grade and the stage timings without token counts or cost, and is recorded as unmeasured.

Then, once the agent finishes a piece of work, ask it:

Take the AMI survey regarding the ticket triage you just did

There is nothing to clone and nothing to keep updated. No token needs to be supplied: the first call that requires one registers the machine and stores the token at ~/.ami-survey/token.

uvx is part of uv, the tool the MCP documentation uses for Python servers. If uv is not installed, GETTING-STARTED.md gives a pipx form and a route that requires neither.

If you already hold a token, set it as AMI_API_TOKEN in that block's env and it will be used instead of registering a new one.

Or use the remote connector, with nothing installed

In claude.ai, open Settings, then Connectors, then Add custom connector, and supply:

https://survey.agentbenchmark.dev/mcp

Nothing is installed and no token is required. These runs are recorded as unmeasured and are never compared against measured ones: a remote server cannot read your runtime's logs, so token counts and cost are absent rather than estimated.

Licence

MIT. Use it, modify it, redistribute it, include it in your own work.

The licence covers this client only. It grants no rights in the survey service the client submits to, or in the data that service collects.

Full terms in LICENSE.

What leaves your computer

Token counts, timings, model names, the stage names the workflow declared, and the grade. Not your files, not your prompts, not your shell commands. GETTING-STARTED.md sets this out in full.

Submissions go to survey.agentbenchmark.dev and nowhere else. That destination is a constant in the source rather than a setting, so a stale environment variable cannot redirect a submission onto your own disk. That is the one failure which would make a run appear successful while collecting nothing.

Requirements

Python 3.9 or newer. No dependencies; the standard library only.

Further reading

  • GETTING-STARTED.md assumes no prior setup, covers macOS, Linux and Windows, and explains what to do when the tools do not appear.

  • COMMANDS.md documents the clone-and-run route, which benchmarks one workflow across several models on your own API key. It is not required in order to take part.

  • MAKE-IT-MEASURABLE.md explains how to structure a workflow so that there is something worth measuring.

Available Tools

14 tools
ami_collect_telemetryA

Measure the workflow run: detects which agent runtime you are in, reads that runtime's own call records (token usage, model, timings reported by the provider), attributes each call to a stage or observed execution phase, and stores the result on the run. Returns the collected inventory fields. Use these numbers verbatim. If your runtime has no adapter, this says so and you should use ami_record_calls instead.

ParametersJSON Schema
NameRequiredDescriptionDefault
cwdNo
run_idNoDefaults to the active run.
adapterNoForce a telemetry adapter instead of detecting the runtime. Rarely needed.
session_idNo
window_endNoISO-8601 override.
window_startNoISO-8601 override.
transcript_pathNo

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With zero annotations, the description carries the full burden, and it does decent work: it discloses the persistence side effect ("stores the result on the run"), the graceful no-adapter failure mode, and the verbatim-numbers rule. But it never states re-invocation semantics (overwrite vs. append), what happens with partial/empty records, or any permission requirements — notable gaps for a tool that writes state.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences, roughly 70 words, with the purpose front-loaded and the fallback instruction last. Every sentence earns its place; the terse operational directive "Use these numbers verbatim" adds real value without bloat.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-param tool with no annotations and no output schema, the description covers the essentials: purpose, mechanism, persistence side effect, return value ("collected inventory fields"), and the adapter fallback. It falls short only on re-invocation behavior and workflow timing, which are refinement-level rather than call-blocking details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 57% (4/7 params documented; cwd, session_id, transcript_path have no descriptions). The description adds useful context — the runtime-detection mechanism clarifies the 'adapter' param, and the provider-reported records (token usage, model, timings) give meaning to the ISO-8601 window overrides — but it never explicitly explains the three undocumented parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Measure the workflow run" states a specific verb plus resource, then elaborates the mechanism (detect runtime, read provider call records, attribute to stages, store on run, return inventory). It explicitly names ami_record_calls as the fallback alternative, cleanly distinguishing itself from the closest sibling among 13 tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The final sentence gives an explicit conditional: "If your runtime has no adapter, this says so and you should use ami_record_calls instead" — naming the exact alternative and the condition that selects it. However, it doesn't clarify workflow ordering relative to stage-marking siblings like ami_mark_stage, nor give broader when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_get_grading_scaleA

Return the AMI output-quality grading scale. Read this before grading: agent_output_grade must be one of its codes and is validated on submit.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It lacks an explicit statement of no side effects or read-only status, though 'Return' suggests a safe getter. However, it adds meaningful normative context: the returned codes are validated at submit time, which signals the agent must treat the scale as authoritative. Missing output format/response shape is a typical gap here.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, arrest in front-loaded action first, second is a specific usage directive. There is no fluff, and every clause contributes (what, why, when).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a no-parameter getter with no output schema, the description covers the essential: what the tool is for and how its output relates to a validation rule. Only principal missing piece is an example or format of the returned scale, but that is not critical for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, so the schema coverage is trivially 100%. The description adds semantic value beyond that by explaining why the tool should be used (to obtain valid codes for agent_output_grade). Baseline 4 for a parameterless tool.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States exactly what it does: returns the AMI output-quality grading scale. This is a specific resource with a specific verb, and no sibling tool overlaps with this responsibility; it's clearly distinct from ami_get_workflow_categories and ami_get_instructions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use guidance: 'Read this before grading' and explains why by noting agent_output_grade must use one of its codes and is validated on submit. It doesn't explicitly name alternatives or exclusions, but none are needed since this is the only source of the grading codes.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_get_instructionsA

Return the AMI survey procedure as markdown. Call this FIRST if your harness has no skill mechanism that already gave you the procedure - it tells you the call order and, for your runtime, how telemetry is obtained. Claude Code agents already have it as a skill and do not need this.

ParametersJSON Schema
NameRequiredDescriptionDefault
runtimeNoYour runtime. 'mcp' (default) for any MCP client that is not Claude Code.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It discloses the output format (markdown), the content (call order and telemetry acquisition), and the tool's role as the initial instruction provider. It does not explicitly state that the call is read-only/idempotent, but the described behavior makes that implicit; no contradictory annotations exist.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, with the primary purpose front-loaded in the first sentence. The second and third sentences earn their place by providing conditional usage and an exclusion. No filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is self-sufficient for an optional one-parameter read tool. It tells the agent what the tool returns, when to call it (and when not to), and how the runtime parameter affects the content. With no output schema needed for a markdown-returning tool, no essential information is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description reinforces the runtime parameter by saying the procedure varies by runtime ('for your runtime, how telemetry is obtained') but does not add concrete parameter-level syntax beyond the enum and default already in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a clear verb ('Return'), resource ('AMI survey procedure'), and format ('as markdown'). It differentiates from siblings by explicitly positioning it as the entry point ('Call this FIRST') for the survey workflow, distinguishing it from the other ami_* tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('if your harness has no skill mechanism that already gave you the procedure') and when-not-to-use ('Claude Code agents already have it as a skill and do not need this'), and even names the alternative mechanism (a skill). Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_get_reportA

Render the human-readable Markdown report for a run (defaults to the active run). With no run and no active run, returns the index of all submitted surveys.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral burden. It discloses defaulting behavior, the no-run fallback, and implies a read-only report operation. It does not cover error cases, but none are implied for this simple getter.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences deliver the core behavior and fallback without repeating schema or annotation information. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter, no-output-schema tool, the description covers the main behavior, the optionality of the run, the active-run selection, and the no-active-run fallback, including the Markdown output format.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The only parameter, run_id, has no schema description. The text adds meaning by explaining that omitting it selects the active runcard and that with no active run an index is returned.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Render') and resource ('human-readable report for a run'), then explains the fallback to the survey index. This makes the tool's purpose unambiguous and distinguishable from sibling report/survey tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly explains when the default active run is used and what happens when no active run exists. It does not name alternative sibling tools, but the usage context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_get_scorecardA

The scorecard for a submitted run: the AMI Maturity Index, the Performance Score, the five pillars, and structured findings. Every number and finding is computed from the run's own data - the server calls no model. If a human wants this read back as prose, write it yourself from narration_brief.findings, following the instructions there.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idNoDefaults to the run you just submitted.

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the transparency burden. It usefully discloses that every value is computed from the run's own data and that the server calls no model, which is important behavioral context for an agent deciding whether to invoke it. It does not cover error cases or side effects, but for a read-only getter the key behavioral risk is addressed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, both high-value. The first front-loads the exact contents of the scorecard; the second provides important behavioral transparency and a practical instruction about prose generation. No filler or redundant restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple, one-optional-parameter getter with no output schema, the description covers what the tool returns and how the data is produced. It lacks a few details like exact field structure or error conditions, but the agent has enough to invoke it correctly and interpret the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents the only parameter, run_id, including its default behavior. The description adds no additional parameter meaning beyond the schema, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool's resource and output: the scorecard for a submitted run, including AMI Maturity Index, Performance Score, five pillars, and structured findings. It does not explicitly differentiate from sibling tools like ami_get_report, but the specific list of outputs makes the purpose fairly unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use the tool: when you need structured scorecard data for a run. It also gives a clear instruction about not using the output as prose and instead writing prose from narration_brief.findings. However, it does not explicitly mention alternatives or when to choose a sibling tool instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_get_surveyA

Return the survey definition: every field from Collection_Inventory.csv, how each one is obtained, and which ones you must answer yourself.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does useful work by disclosing what the tool returns and the actionable instruction about which fields the user must answer. It clearly frames this as a retrieval operation, though it does not explicitly say there are no side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence that is front-loaded with the main action and includes the key details of the returned content. There is no filler or redundancy; every clause adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter tool with no annotations and no output schema, the description is reasonably complete: it states what is returned and what the user should do with it. It could be slightly stronger by noting that this is a read-only operation or how it relates to the survey workflow, but the core calling context is covered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the baseline is 4. The description need not explain parameter semantics, and the empty schema is unambiguous. No additional parameter information is required.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a clear verb ('Return') and specifies the resource ('the survey definition') along with its contents. It distinguishes itself from siblings by mentioning fields and how they are obtained, though the reference to 'Collection_Inventory.csv' assumes context and could be more self-contained.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit guidance on when to use this tool versus siblings such as ami_get_instructions, ami_survey_begin, or ami_survey_status. The intended use is implied—obtain survey definition before answering—but no alternatives, exclusions, or prerequisites are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_get_workflow_categoriesA

The workflow categories a run may declare itself into, and what each one covers. A category decides which other workflows this run is compared against, so read the list rather than inventing a label.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It implies a read-only retrieval but doesn't explicitly state side-effect freedom or any behavioral traits like idempotency or error conditions. It does add useful context about the meaning of categories, which goes beyond a simple 'get' but lacks explicit behavioral transparency, so a 3 is warranted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with zero waste. The first sentence states the resource and its meaning; the second gives actionable advice. Every word earns its place, and the structure is front-loaded with the core definition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless retrieval with no output schema, the description conveys the content (categories and their coverage) and the purpose (comparison selection). It could be more explicit about the exact return format, but the implication that it returns a list with descriptions is clear. Overall, it is sufficiently complete for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, so the baseline is 4. The schema coverage is effectively 100% (no properties), and the description correctly avoids mentioning parameters. No additional parameter semantics are needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool's purpose: retrieving workflow categories and explaining what each covers. It ties the categories to the comparison logic, which distinguishes it from other get tools by topic, though it doesn't explicitly name sibling alternatives. This is a clear, specific purpose, but lacks explicit differentiation, so a 4 is appropriate.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides an explicit usage directive: 'read the list rather than inventing a label.' This tells the agent to consult the tool before declaring a category, which is strong usage guidance. It doesn't mention when not to use it or alternatives, but the core when-to-use is clear, earning a 4.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_list_surveysA

List all submitted survey responses and where they are stored.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It implies a read-only list operation but does not explicitly state that it is non-destructive, nor does it mention potential side effects, return format, ordering, pagination, or any additional behavior. For a simple list, the lack of explicit read-only assurance is a notable gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, efficient sentence that immediately states the action and scope. No wasted words, and the core information is front-loaded. This is an exemplary concise structure.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless list tool, the description covers the essential purpose but omits behavioral context like whether it returns a count, supports filtering, or how 'where they are stored' is represented. Given no output schema and no annotations, the description is minimally adequate but not complete. It is sufficient for a trivial tool, but leaves room for clarification about the return value shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema trivially covers 100% of parameter documentation. Baseline for 0 params is 4; the description adds no parameter-specific meaning, which is acceptable because there are none to describe. It does not detract from clarity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action ('List'), a clear resource ('all submitted survey responses'), and an additional expectation ('where they are stored'). It is unambiguous and implicitly distinguishes itself from sibling tools like ami_get_survey (single survey) and ami_submit_survey (submission) by focusing on listing all responses.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides no explicit guidance on when to use this tool versus alternatives. While 'all' implies a bulk operation, there is no mention of exclusions, prerequisites, or comparisons with other survey-related tools. The agent must infer usage from the name and description alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_mark_stageA

Declare the workflow stage you are entering, e.g. 'Classify Severity' or 'Draft Customer Reply'. Call it as you move through the workflow to get a declared-stage Agent Effort Profile; without markers the profile falls back to AMI-observed execution phases. No survey run is needed first: markers emitted before ami_survey_begin are buffered with the timestamp you emitted them at and attached to the run when it opens. When the last stage is done, call once more with closes=true - each marker ends the stage before it, so without a closing one the final stage runs to the end of the measurement window and absorbs everything you do afterwards.

ParametersJSON Schema
NameRequiredDescriptionDefault
cwdNoOptional workspace override; buffered markers are adopted by a run opened in the same workspace.
noteNo
stageNoName of the stage being entered.
closesNoTrue to end declared work rather than start a stage. Call this when the final stage is complete, before you verify output or report back; work after it is attributed to observed phases instead of to a stage.
run_idNo
marked_atNoISO-8601; defaults to now.

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description compensates by explaining important behavior: markers are buffered before a survey run exists, they are attached to runs in the same workspace, and an unclosed final stage absorbs subsequent work. This is substantial behavioral disclosure beyond a simple one-line purpose.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the tool's purpose and then covers invocation timing and closing behavior without fluff. The buffering sentence is long but still readable and every sentence contributes necessary usage information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the full intended lifecycle—opening, buffering, workspace adoption, and closing—and explains the main pitfall of forgetting to close the final stage. Minor gaps around note and run_id remain, but they are not essential for correct selection and basic invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers 4 of 6 parameters, and the description adds meaningful semantics for stage and closes, including lifecycle edge cases. However, note and run_id remain essentially undocumented, and the connection between cwd and buffered markers is only implied.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose—'Declare the workflow stage you are entering'—with concrete examples and ties it to the Agent Effort Profile. It is distinct from sibling tools like ami_survey_begin or ami_record_calls by focusing on stage markers.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit guidance on when to call the tool ('as you move through the workflow'), when it can be called relative to ami_survey_begin, and how to close the final stage with closes=true. It does not explicitly name alternative tools or say when not to use it, but the workflow context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_record_callsA

Telemetry entry point for agents NOT running in Claude Code. Post the provider-reported usage for each API call the workflow made. Every record needs model, start_time, end_time, input_tokens, output_tokens - values read from real API responses, never estimated.

ParametersJSON Schema
NameRequiredDescriptionDefault
callsYesCall records.
run_idNo
adapterNoName of the runtime/SDK the records came from.
replaceNo
runtime_metadataNoe.g. {platform, runtime, version}.

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden. It does add a strong data-quality rule: 'values read from real API responses, never estimated.' Yet it doesn't disclose the nature of the write (post) or side effects, what replace means, whether this is append-only, or any auth/rate-limit implications. It's not misleading, but the description only partially covers behavioral transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with zero filler. The scoping ('agents NOT running in Claude Code') is front-loaded, the purpose ('post usage') is explicit, and the data-quality rule ('never estimated') earns its place. This is an efficiently structured description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 5 parameters, a nested object, no output schema, and no annotations, so the description must carry more explanatory weight. It provides the required-field rule and sourcing constraint, but leaves the replace parameter's semantics unexplained, never hints whether records append or overwrite, and does not clarify what the tool returns or whether the call is idempotent. These gaps make it adequate but incomplete for an agent to use confidently without more context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 60% (6 out of 10 properties have descriptions, but several important ones like replace, run_id, and some fields inside nested objects lack descriptions). The description adds 'never estimated' semantic guidance for input_tokens/output_tokens, but doesn't explain replace, run_id, or how these interact. It does not fully compensate for the uncovered schema areas, but the baseline 3 is appropriate given partial coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('post') and resource ('provider-reported usage for each API call'), and claims an explicit scope ('agents NOT running in Claude Code'). It does not name a sibling alternative (e.g., ami_collect_telemetry) but the distinction is understandable in context.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives a clear when-to-use condition: agents NOT running in Claude Code. This effectively says 'when you're not in Claude Code, use this,' though it does not explicitly name the alternative/does-not-use case (e.g., Claude Code agents should use something else). It provides a meaningful exclusion but not explicit sibling routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_submit_surveyA

Submit and persist the survey. Requires a grade from the AMI grading scale, a justification, and evidence (the concrete artifacts produced). Writes JSON + Markdown + a CSV index row to disk and returns the paths.

ParametersJSON Schema
NameRequiredDescriptionDefault
graderNo'self' when the agent grades its own output, 'human' or 'external_reviewer' when a person supplied the grade.
run_idNo
workflow_nameNo
grade_evidenceYesConcrete artifacts being graded: file paths, ticket ids, message ids, tool outputs.
workflow_end_timeNo
agent_output_gradeYesA grade code from ami_get_grading_scale.
grade_justificationYesWhy that grade, measured against the workflow's stated requirements. At least 40 characters.
workflow_descriptionNo
allow_empty_telemetryNoOnly for deliberately unmeasured runs; measurement fields will be null.

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It explicitly discloses side effects: 'Writes JSON + Markdown + a CSV index row to disk and returns the paths.' This is significant behavioral transparency. It does not cover idempotency, failure modes, or overwrite behavior, but the key traits are stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no filler. Purpose and side effects are front-loaded, making it easy for an agent to quickly understand what the tool does and what it returns.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 9 parameters, no output schema, and no annotations. The description covers purpose, required inputs, side effects, and return paths, but lacks detail on optional parameters, validation rules, and error handling. For a submission tool with many parameters, more context would help, but the essentials are present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 56%, meaning 5 of 9 parameters have descriptions. The description reiterates the three required fields (grade, justification, evidence) but adds no new syntax or formatting details. It does not explain optional parameters like grader, run_id, workflow_name, or allow_empty_telemetry. It provides some added meaning for required fields but does not fully compensate for the coverage gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Submit and persist the survey.' It uses a specific verb (submit/persist) and resource (survey), and mentions side effects (writes files, returns paths). This distinguishes it from siblings like ami_survey_begin or ami_get_survey.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by stating required inputs (grade, justification, evidence), suggesting it is the final step after grading. However, it does not explicitly state when to use it vs. other survey tools or mention any exclusions. Sibling tools like ami_survey_begin, ami_survey_status, and ami_get_survey exist, but no direct comparison is made.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_survey_beginA

Open an AMI survey for the workflow you have just completed. Detects the runtime, session and measurement window automatically, and adopts any stage markers you buffered with ami_mark_stage while working. Call this FIRST, after the workflow's real work is finished, so the survey's own token spend is excluded from the measurements.

ParametersJSON Schema
NameRequiredDescriptionDefault
cwdNoOptional working-directory override.
adapterNoForce a telemetry adapter instead of detecting the runtime. Rarely needed.
work_unitNoOptional, but declare it together with work_unit_count. The countable thing this workflow handled: 'ticket', 'CV', 'support email'. It is what cost and duration get divided by, so a workflow that did more work is not penalised for costing more.
session_idNoOptional session id override.
workflow_nameYesShort reusable label for the workflow, e.g. 'Support Ticket Triage & Response'.
transcript_pathNoOptional explicit transcript path override.
work_unit_countNoOptional, but declare it together with work_unit. How many work units this run actually handled - a whole number you can point at in the output, not an estimate.
workflow_categoryNoOptional. The workflow's category, from ami_get_workflow_categories. Decides which other workflows this run is compared against; an undeclared workflow is only ever compared against itself. Take it from the workflow's own workflow.json where one exists - do not invent one.
workflow_end_timeNoOptional ISO-8601 override for when the workflow ended.
workflow_start_timeNoOptional ISO-8601 override for when the workflow began. Defaults to the first human turn of this session.
workflow_descriptionYes1-3 sentences: what the workflow was given and what business work it produced. Minimum 20 characters.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it uses that space well: it discloses automatic runtime/session/window detection, adoption of buffered stage markers, and the token-spend exclusion. It does not cover repeated-call behavior or what a returned survey handle looks like, but the essential behavior is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, each earning its place: purpose, automatic behavior, and invocation timing/why. The key action and timing are front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a survey-begin tool with no output schema, the description gives enough to call it correctly: when to call it, what it auto-detects, and how it relates to ami_mark_stage. A short note on what the call returns would make it fully complete, but no critical invocation detail is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 11 parameters in detail. The description does not add extra parameter nuance, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Open an AMI survey for the workflow you have just completed.' It distinguishes this from sibling survey tools by calling it the FIRST step after the workflow's real work, which separates it from ami_submit_survey, ami_survey_status, and ami_get_survey.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit positional instruction: 'Call this FIRST, after the workflow's real work is finished.' It also explains the measurement rationale (excluding the survey's own token spend). It does not name alternatives or list exclusion cases, so it falls just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_survey_statusA

Show the current values collected for the active run, which inventory fields are still empty, and what is blocking submission.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idNo

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It communicates that this is a read-only-ish status check via 'Show', but it does not explicitly state that no data is modified, nor does it cover potential requirements like having an active run or error behavior. The description adds no safety or side-effect disclosure beyond the word 'Show'.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence efficiently communicates the resource and trims outputs, with no fluff. Every clause adds information and the description is easy to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter tool with no output schema and no annotations, this is largely complete: it names all three output CPU dimensions. The gaps are small, mainly missing explicit run_id shape and any prerequisite such as a 'defined' active run.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only declares run_id as a string with 0% coverage, so the description must compensate. 'For the active run' hints that run_id identifies a current run, but the description does not explicitly explain run_id's meaning, whether it is optional, or how the 'active run' is determined.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Show') with a clear resource ('current values collected for the active run') and elaborates the three key outputs: collected values, empty fields, and submission blockers. This distinguishes it from sibling tools like ami_get_survey (likely returns survey questions) and ami_submit_survey (performs the submission).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The mention of 'what is blocking submission' implies this is for pre-submission status checks, but the description does not explicitly state when to use this tool versus alternatives such as ami_get_survey or ami_submit_survey. There is no exclusionary or 'when not to use' guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ami_write_findingsA

Write the judgement sections of a run's scorecard. The server computes every number and the three sections that follow from them; these four are readings of the work that no arithmetic produces, so they are yours to write. Call ami_get_scorecard first and use narration_brief.sections_awaiting_you - it carries the brief for each. Ground every sentence in the run's own evidence; do not invent industry context you were not given.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe submitted run.
key_findingNoOptional. Overrides the derived summary if you have read the output and know better than the arithmetic does.
industry_next_stepNoOne thing the business should decide or standardise.
workflow_next_stepNoOne concrete change, specific enough to act on this week.
industry_opportunityNoWhat this workflow being agent-run means commercially. Say so plainly if you were given no industry context.
workflow_opportunityNoThe single biggest improvement available to this workflow. Name the stage.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden, and it does so well: it discloses what the server computes vs. what the agent supplies, and imposes a grounding constraint ('Ground every sentence in the run's own evidence; do not invent industry context'). That is exactly the kind of non-obvious behavior an agent needs to know.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, each earning its place: purpose, prerequisite, and grounding constraint. Front-loaded and no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a write tool with no annotations and no output schema, it explains the division of labor, the prerequisite read, and the evidence constraint. It doesn't mention failure modes or confirmation, but the essentials are covered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds the concept of narration_brief.sections_awaiting_you but not per-parameter nuance; it does not describe how the five optional fields map to those sections. Schema handles the individual parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: 'Write the judgement sections of a run's scorecard.' It clearly distinguishes these from server-computed numbers and sections, and it names the exact prerequisite call (ami_get_scorecard). This makes the tool's purpose unmistakable and distinct from sibling tools like ami_get_scorecard.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly instructs the agent to call ami_get_scorecard first and use narration_brief.sections_awaiting_you, which gives clear context for when to use this tool. It does not explicitly list when not to use it or name alternative tools, but the prerequisite and task boundary are clear enough.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 14 tool updatesv0.1.0
    • First observedami_collect_telemetry
    • First observedami_get_grading_scale
    • First observedami_get_instructions
    • First observedami_get_report
    • First observedami_get_scorecard
    • First observedami_get_survey
    • First observedami_get_workflow_categories
    • First observedami_list_surveys
    • First observedami_mark_stage
    • First observedami_record_calls
    • First observedami_submit_survey
    • First observedami_survey_begin
    • First observedami_survey_status
    • First observedami_write_findings

TDQS

A3.9/5.0

Scored across 14 tools

Disambiguation4/5

Tools are mostly distinct: telemetry has two entry points (ami_collect_telemetry vs ami_record_calls) that could be confused, but descriptions clearly separate by runtime. Other tools target distinct actions (begin, mark, submit, status, etc.).

Naming Consistency4/5

All tools follow the 'ami_' prefix plus clear verb_noun pattern (e.g., ami_get_scorecard, ami_write_findings, ami_submit_survey). Minor deviation: ami_survey_begin uses noun_verb instead of strict verb_noun, but overall consistent.

Tool Count5/5

14 tools is well within the sweet spot for a domain as complex as survey instrumentation and reporting. Each tool serves a distinct lifecycle step (setup, telemetry, grading, submission, reporting), and none feel redundant.

Completeness4/5

The tool surface covers the full survey lifecycle: begin, collect telemetry, mark stages, get instructions/scale/survey, check status, write findings, submit, and retrieve reports/lists. Minor gap: no explicit tool for updating or amending a run after submission, but that is not core to the stated purpose.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers