Skip to main content
Glama

@autousers/mcp

Evaluate UX with AI personas and human raters — directly from Claude, Cursor, ChatGPT, and any MCP-aware client.

npm version MIT licensed Provenance


Install

The fastest way to connect is via the remote server — no package to install, OAuth handles auth automatically.

Paste https://mcp.autousers.ai/mcp into your client's MCP connector UI or config. OAuth 2.1 launches in the browser on first use.

Client

How to connect

Claude.ai

Settings → Connectors → Add custom connector → paste URL

Claude Desktop

Settings → Connectors → Add custom connector → paste URL

Cursor

See config below or use the one-click deeplink

VS Code + Copilot

See config below

ChatGPT

Settings → Connectors → Developer Mode → Add connector → paste URL

Codex CLI

See config below

Cursor~/.cursor/mcp.json:

{
  "mcpServers": {
    "autousers": {
      "url": "https://mcp.autousers.ai/mcp"
    }
  }
}

VS Code + GitHub Copilot.vscode/mcp.json (workspace) or ~/.config/Code/User/mcp.json (global):

{
  "servers": {
    "autousers": {
      "type": "http",
      "url": "https://mcp.autousers.ai/mcp"
    }
  }
}

Claude Code CLI:

claude mcp add --transport http autousers https://mcp.autousers.ai/mcp

Codex CLI:

codex mcp add autousers --url https://mcp.autousers.ai/mcp

Bridge fallback (stdio-only clients)

Clients that only support stdio — Cline, Zed, Continue, Goose — use mcp-remote as a shim:

{
  "mcpServers": {
    "autousers": {
      "command": "npx",
      "args": ["-y", "mcp-remote@>=0.1.16", "https://mcp.autousers.ai/mcp"]
    }
  }
}

Zed~/.config/zed/settings.json:

{
  "context_servers": {
    "autousers": {
      "command": {
        "path": "npx",
        "args": ["-y", "mcp-remote@>=0.1.16", "https://mcp.autousers.ai/mcp"]
      }
    }
  }
}

Continue~/.continue/config.yaml:

mcpServers:
  - name: autousers
    command: npx
    args:
      - -y
      - "mcp-remote@>=0.1.16"
      - https://mcp.autousers.ai/mcp

Stdio / headless / CI

For environments without a browser (CI pipelines, scripted workflows), use the npm package with an API key:

# Claude Code with Bearer token
claude mcp add --transport http autousers https://mcp.autousers.ai/mcp \
  --header "Authorization: Bearer $AUTOUSERS_API_KEY"

Or run as a local stdio process:

npx -y @autousers/mcp

With environment variables:

{
  "mcpServers": {
    "autousers": {
      "command": "npx",
      "args": ["-y", "@autousers/mcp"],
      "env": {
        "AUTOUSERS_API_KEY": "ak_live_..."
      }
    }
  }
}

Related MCP server: crowd-test-mcp

Tools

39 tools across four categories. Read-only tools carry readOnlyHint=true and are safe to call without side effects.

Templates (6)

Tool

Description

templates_list

List question templates available to your team

templates_get

Fetch a single template by ID

templates_create

Create a new team-scoped template

templates_update

Patch a template (only supplied fields change)

templates_delete

Hard-delete a template

templates_duplicate

Deep-clone a template into a destination team

Evaluations (14)

Tool

Description

evaluations_list

List evaluations the caller can see

evaluations_get

Fetch one evaluation including config and comparisons

evaluations_create

Create an SSE or SxS evaluation; optionally queue autouser runs

evaluations_update

Patch fields on an evaluation

evaluations_delete

Delete an evaluation and its dependent rows

evaluations_save_draft

Merge wizard fields into a draft evaluation

evaluations_ratings_list

List human + autouser ratings

evaluations_results_get

Aggregate stats and per-rater summaries

evaluations_agreement_get

Pairwise Cohen's Kappa inter-rater agreement

evaluations_ai_insights_get

AI-authored summary, key findings, and recommendations

evaluations_export_get

Download results as JSON or CSV

evaluations_share_create

Grant a per-user VIEWER / EDITOR / OWNER share

evaluations_shares_list

List explicit per-user shares

evaluations_transfer

Transfer evaluation ownership to another user

Autousers (15)

Tool

Description

autousers_list

List autousers (built-in + custom)

autousers_get

Fetch a single autouser by ID

autousers_create

Create a team-scoped custom autouser

autousers_update

Patch a custom autouser

autousers_delete

Soft-delete a custom autouser

autousers_duplicate

Deep-clone an autouser into a team

autousers_run

Queue autouser runs against an evaluation

autousers_run_stop

Cancel pending or running autouser runs

autouser_status_get

Run statuses and summary counts for an evaluation

autouser_run_get

Fetch one autouser run with full context

autouser_run_turns_list

Per-turn token and cost telemetry for a run

autousers_calibration_start

Compute Cohen's Kappa vs human ratings

autousers_calibration_status_get

Get calibration status

autousers_calibration_freeze

Freeze a rubric version and set it as active

autousers_calibration_optimize

Send disagreements to AI for rubric suggestions

Settings (4)

Tool

Description

teams_list

List every team the caller belongs to

teams_get

Fetch one team's detail

team_members_list

List team members with roles and profiles

usage_get

Usage rollup: free-run pool remaining and token spend


Resources & Prompts

Resources

Three readable resource URIs:

URI

Description

autousers://evaluation/{id}

Full evaluation object including results

autousers://template/{id}

Template definition with all dimensions

autousers://autouser/{id}

Autouser persona with rubric

Prompts

Five canned workflows registered as MCP prompts:

Prompt

Description

evaluate-url

Create an SSE evaluation against a URL and surface results

compare-designs

Create an SxS evaluation between two URLs

analyze-results

Summarise evaluation results with key findings

calibrate-autouser

Run calibration and freeze the rubric when stable

triage-low-agreement

Surface autouser/human disagreements and suggest fixes


Authentication

When you add the server URL to any supported client, an OAuth 2.1 PKCE + DCR flow launches in the browser. After approving, the client receives a short-lived access token (~15 min) that rotates automatically via a refresh token. No static credentials are stored on the client.

Best for: Claude.ai, Claude Desktop, Cursor, VS Code, ChatGPT — any interactive session.

API keys

API keys are long-lived bearer tokens for headless environments. Pass them as Authorization: Bearer ak_live_... — either as an HTTP header in the direct-URL clients or as the AUTOUSERS_API_KEY env var for stdio.

Best for: CI pipelines, Claude Code with --header, Codex CLI --bearer-token, scripts.

Mint keys at app.autousers.ai/settings/api-keys. Keys are shown once at creation — store them in a secrets manager immediately.

Scopes

Scope

Grants

templates:read

List and fetch templates

templates:write

Create, update, delete, duplicate templates

evaluations:read

List, fetch, export, view results

evaluations:write

Create, update, delete, share evaluations

autousers:read

List autousers, fetch runs and telemetry

autousers:write

Create, update, delete, run, calibrate autousers

ratings:read

List ratings for an evaluation


Configuration

Variable

Default

Description

AUTOUSERS_API_KEY

Bearer token for headless / stdio auth (ak_live_...)

AUTOUSERS_BASE_URL

https://app.autousers.ai

Override the API host (e.g. for self-hosted or local dev)

The default base URL points to production. You only need AUTOUSERS_BASE_URL if you are targeting a different environment.


Diagnostics

Run the bundled doctor command to verify your configuration:

npx -y -p @autousers/mcp autousers-mcp-doctor

It checks that AUTOUSERS_API_KEY is set (for stdio auth), that the API host is reachable, and that at least one tool call succeeds. Exits non-zero on any misconfiguration so it integrates cleanly into CI pre-flight checks.


Available Tools

39 tools
autouser_run_getGet a single autouser runA
Read-onlyIdempotent

Fetch one autouser run with autouser+evaluation summaries. The response includes a viewUrl deep-link to open the run (with session replay) directly in the Autousers app. Example: { evaluationId, runId }.

ParametersJSON Schema
NameRequiredDescriptionDefault
evaluationIdYesEvaluation id (URL path).
runIdYesAutouser run id (URL path).

Output Schema

ParametersJSON Schema
NameRequiredDescription
viewUrlNo

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, non-destructive behavior. The description adds value by disclosing that the response includes a viewUrl deep-link and summaries, which are behavioral traits beyond the schema. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no wasted words. The description is front-loaded with the core action and resource, followed by key additional information (deep-link) and an example. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema and comprehensive annotations, the description is fully adequate. It covers the purpose, key response feature (viewUrl), and provides an example, leaving no critical gaps for a simple fetch tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% coverage with basic descriptions ('Evaluation id (URL path).'). The description does not add further semantic details about the parameters beyond the example usage, so it meets but does not exceed the baseline for high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Fetch one autouser run') and the resource ('with autouser+evaluation summaries'). It distinguishes from sibling tools like autousers_run (create) and autouser_run_turns_list (list turns) by specifying it retrieves a single run with summaries and a deep-link.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when you have an evaluationId and runId to retrieve run details, but it does not explicitly state when to use this tool vs alternatives (e.g., autouser_status_get for status) or provide exclusions. The example hints at typical use, but no direct guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autouser_run_turns_listList per-turn telemetry for an autouser runB
Read-onlyIdempotent

Return per-turn token+cost telemetry for a run. Example: { evaluationId, runId }.

ParametersJSON Schema
NameRequiredDescriptionDefault
evaluationIdYesEvaluation id (URL path).
runIdYesAutouser run id (URL path).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate the tool is read-only, non-destructive, and idempotent. The description adds that it returns per-turn telemetry, which is consistent but adds limited new behavioral context beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: one sentence plus an example. It is front-loaded and every element is relevant, with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with two parameters, the description is adequate. However, it lacks information about pagination, limits, or ordering, which would be useful for an agent to use the tool correctly. The existence of an output schema partially compensates.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% coverage with descriptions for both parameters. The description shows an example usage, which is helpful but does not add new semantic meaning beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that the tool returns per-turn token+cost telemetry for a run, with a specific verb and resource. However, it does not explicitly distinguish it from sibling tools like autouser_run_get, which might return run-level data.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It only gives an example of parameters but does not mention exclusions or context for usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_calibration_freezeFreeze an autouser rubric versionA
Idempotent

Mark a rubric as frozen and set it as active. Example: { id: 'au_1', commitMessage: 'v3 final' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id.
rubricIdNoOptional: rubric version id to freeze; defaults to current activeRubricId.
commitMessageNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds the context of freezing and setting active, which is consistent with annotations (readOnlyHint=false, idempotentHint=true). No additional behavioral details beyond what annotations convey, but no contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: one sentence and a relevant example. No unnecessary words, and the example aids clarity. It is front-loaded with the action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that the tool has three parameters (one required) and an output schema, the description is too brief. It does not explain the implications of freezing, such as whether it prevents further modifications or affects other calibration operations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 67%, meaning the 'commitMessage' parameter lacks description. The tool description does not fill this gap; it only provides an example. This adds minimal semantic value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('freeze and set as active') and the resource ('rubric'). It distinguishes itself from sibling calibration tools like start or optimize, which perform different operations.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not explicitly provide when-to-use or when-not-to-use guidance. It implies usage through the example but lacks clarity on prerequisites or alternatives among sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_calibration_optimizeOptimize autouser rubric from disagreementsC

Send disagreements to Gemini for rubric suggestions. Example: { id, disagreements: [{ ratingId, humanReasoning }] }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id.
disagreementsYesDisagreement payload sent to Gemini for rubric suggestions.
manualRubricEditNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description states it sends data to Gemini, implying an external side effect, but provides no details on potential consequences (e.g., cost, latency, data handling). Annotations declare readOnlyHint=false and destructiveHint=false, but the description does not elaborate on behavioral traits beyond the bare action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short (one sentence plus a JSON example), but the example is minimal and not fully integrated. Every sentence adds some value, but the structure could be improved by merging the example into a clearer explanation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (3 parameters, output schema exists), the description is too terse. It does not explain what the tool returns or how rubric suggestions are formatted, leaving the agent to guess.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 67%; the description adds an example for the 'disagreements' array, illustrating the structure beyond the schema. However, the 'manualRubricEdit' parameter lacks any description or example, leaving a gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Send disagreements') and the outcome ('rubric suggestions') using a specific verb and resource. It distinguishes from sibling tools like 'autousers_calibration_freeze' and 'autousers_calibration_start'. However, the purpose could be more precise by explaining what generating rubric suggestions entails.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives like 'autousers_calibration_freeze' or 'autousers_calibration_start'. There is no mention of prerequisites, when not to use, or recommended scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_calibration_startStart autouser calibration against an evaluationB

Compute Cohen's Kappa vs human ratings. Example: { id: 'au_1', evaluationId: 'ev_2' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id.
evaluationIdYesEvaluation id whose human ratings drive the kappa calc.
rubricIdNoOptional: specific rubric version id; defaults to active.
sampleSizeNo
rubricOverrideNoOptional: human-edited rubric text to grade against (persisted as new version).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond annotations (readOnlyHint=false, destructiveHint=false), the description adds 'Compute Cohen's Kappa' but does not disclose side effects, state changes, or potential async behavior. The example does not substitute for behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Very concise (one sentence + example), but could be slightly more descriptive without verbosity. No wasted words, but lacks depth.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having an output schema, the description omits return values or behavior. No mention of whether the operation is synchronous or asynchronous, or its relation to sibling calibration tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 80%, but the description only provides an example, not additional meaning. No elaboration on parameters like rubricId, sampleSize, or rubricOverride beyond their schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states 'Compute Cohen's Kappa vs human ratings' with a specific verb and resource. Distinguishes from sibling calibration tools (freeze, optimize, status_get).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage to start calibration for an autouser against an evaluation, but does not specify when to use vs. alternatives (e.g., when to start vs. optimize). No exclusion criteria or chaining guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_calibration_status_getGet autouser calibration statusA
Read-onlyIdempotent

Return calibration status; pass includeEvals to also list pickable evals. Example: { id: 'au_1' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id.
includeEvalsNoWhen true, also return availableEvals[] for the wizard's eval picker.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true. The description adds no additional behavioral context beyond what annotations provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose, example, and all necessary information with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema and comprehensive annotations, the description covers the tool's purpose and parameter usage fully. No gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers 100% of parameters with descriptions. Description adds an example showing the id format, which adds value beyond the schema's type information.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the tool returns calibration status and optionally includes available evals. Distinguishes from sibling calibration tools that freeze, optimize, or start.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use versus alternatives. The purpose is clear, but a statement about when not to use (e.g., if you need to modify calibration) would help.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_createCreate a custom autouserA

Create a team-scoped custom autouser. Example: { teamId, name: 'Skeptic', systemPrompt: 'You are...' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
teamIdYesOwning team id (caller must be Editor+).
nameYesDisplay name for the persona.
descriptionNo
roleNoShort role label, defaults to 'autouser'.
avatarNo
systemPromptYesSystem prompt fed to Gemini at run time.
statusNoDefaults to 'published'.
visibilityNoDefaults to 'private'.
capabilitiesNoPersona metadata bundle (stored in `capabilities` JSON column).
configNoRuntime model config (stored in `config` JSON column).

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
roleNo
descriptionNo
statusNo
visibilityNo
isSystemNo
sourceNo
capabilitiesNo
configNo
createdAtNo
updatedAtNo

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description indicates a create operation, which is consistent with annotations (readOnlyHint=false). However, it does not disclose additional behavioral traits such as side effects, required permissions (though partially hinted in parameter descriptions), or rate limits. The annotations already carry the safety profile, and the description adds minimal extra context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is exceptionally concise: one sentence and an example. It front-loads the core purpose and includes necessary context without any redundant or wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (10 parameters, nested objects) and the existence of an output schema, the description is sufficient for an AI to understand the basic function. It could be more complete by mentioning the return value, but the output schema likely covers that. The description is adequate for selection among siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 80% schema description coverage, the baseline is 3. The description adds value by providing a concrete example (teamId, name, systemPrompt), which aids understanding of parameter usage. However, it does not further explain the semantics of nested objects like capabilities or config beyond what is in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool creates a team-scoped custom autouser, using a specific verb and resource. The example further clarifies the inputs. This distinguishes it from sibling tools like autousers_update or autousers_delete, as it is the creation operation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is used when a new custom autouser needs to be created within a team. However, it does not explicitly provide guidance on when to use this tool versus alternatives like autousers_duplicate, nor does it specify prerequisites or situations where it should not be used.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_deleteDelete an autouserA
DestructiveIdempotent

Soft-delete a custom autouser (Admin+ on team). Example: { id: 'au_123' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
deletedYes
idYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds 'soft-delete' nuance and permission level beyond annotations. Annotations already indicate destructive and idempotent hints; description complements them without contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: action+permission then example. No wasted words, front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With output schema present, return values need not be explained. Describes soft-delete and permission; could mention behavior on non-existent id but idempotent hint makes that less critical.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema covers parameter fully (100% description coverage) but description adds an example id format 'au_123', providing extra context beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states verb 'soft-delete' and resource 'custom autouser'. Distinguishes from siblings like create, update, and list tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Specifies team role 'Admin+' but does not explicitly state when to use versus alternatives or when not to use. Adequate but lacks comparative guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_duplicateDuplicate an autouserA

Deep-clone a visible autouser into a team. Example: { id: 'au_123', teamId: 'team_456' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesSource autouser id to clone.
teamIdNoDestination team id; defaults to the caller's active team.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
roleNo
descriptionNo
statusNo
visibilityNo
isSystemNo
sourceNo
capabilitiesNo
configNo
createdAtNo
updatedAtNo

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate mutation (readOnlyHint=false) and non-destructive hint; description adds 'deep-clone' implying full copy. However, it does not clarify if source remains unchanged or mention any constraints beyond visibility.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely concise: one sentence and an example. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema covers return values. Description mentions constraint ('visible') and default for teamId. But missing details on what 'deep-clone' includes/excludes. Nearly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema already describes both parameters with 100% coverage. Description adds only an example, no new semantics beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('deep-clone') and resource ('visible autouser into a team'), and the example reinforces it. It distinguishes from siblings like autousers_create and templates_duplicate.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives (e.g., autousers_create vs. duplicate). No when-not-to-use or prerequisites are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_getGet an autouser by IDA
Read-onlyIdempotent

Fetch a single autouser by id. Example: id from autousers_list.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
roleNo
descriptionNo
statusNo
visibilityNo
isSystemNo
sourceNo
capabilitiesNo
configNo
createdAtNo
updatedAtNo

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true. Description adds that it fetches by id but adds no extra behavioral detail (e.g., error handling). No contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded, no unnecessary words. Efficient and clear.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple fetch tool with complete annotations and output schema, description is adequate. Could mention that it returns the autouser object, but output schema covers that. Nearly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage 100% with description 'Autouser id.' Description adds example 'id from autousers_list', providing context but not deeper semantics. Baseline 3 with slight improvement.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states 'Fetch a single autouser by id', specifying verb and resource. Differentiates from siblings like autousers_list (list multiple) and autousers_create (create).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit example 'id from autousers_list', guiding when to use. Could be improved by noting not to use for listing or other operations, but context from siblings suffices.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_listList autousersA
Read-onlyIdempotent

List autousers (synthetic personas) available to the caller. By default this returns BOTH the canonical built-in personas (e.g. 'Power User', 'Casual User' — every account sees these) AND the caller's custom team-owned autousers. Each row carries a source field ('built-in' or 'custom') and an isSystem boolean to disambiguate; built-ins are NOT duplicates of custom autousers even when names overlap. Pass includeSystem:false to hide built-ins. Pass teamId to scope to one team. Each id appears exactly once. Example: { includeSystem: false, limit: 50 } for 'show me only the autousers I have created'.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMax items to return (1-100, default 20).
teamIdNoOptional: restrict to autousers owned by this team. Caller must be a Viewer+ on that team.
includeSystemNoInclude built-in (system) autousers in the list. Defaults to true. Set to false to see only custom team-owned autousers — useful when a caller asks 'what autousers have I created' and you do not want the canonical built-ins ('Power User', 'Casual User', etc.) to crowd the response.
visibilityNoOptional: filter to public or private autousers only.
statusNoOptional: filter by autouser status.

Output Schema

ParametersJSON Schema
NameRequiredDescription
dataYes
has_moreNo
next_cursorNo

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate a safe read operation (readOnlyHint, destructiveHint false, idempotent). The description adds valuable behavioral context: default returns both built-in and custom with disambiguation via 'source' and 'isSystem' fields, explains that built-ins are not duplicates, and notes that each id appears exactly once. It also mentions the permission requirement for teamId. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is fairly long (5 sentences) but each sentence serves a purpose: stating the primary function, explaining default behavior, disambiguation, filtering options, and an example. It is front-loaded and efficient, though slightly wordy in parts.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the 5 optional parameters and an output schema (not shown but referenced), the description covers default behavior, filtering, disambiguation, and permissions. It explains the source field and isSystem boolean, which are key to understanding the output. No obvious gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds value beyond the schema by providing usage context: explaining the effect of includeSystem, giving an example, and clarifying when to use each filter. This elevates the score above baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'List autousers (synthetic personas) available to the caller.' It distinguishes this from sibling tools like autousers_get (single) and autousers_create by focusing on listing and filtering. The explanation of returned types (built-in vs custom) adds specificity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use filters: 'Pass includeSystem:false to hide built-ins' and 'Pass teamId to scope to one team.' It includes an example that ties to a common use case ('show me only the autousers I have created'). However, it does not explicitly state when not to use this tool or compare it to siblings like autousers_run.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_runRun autousers on an evaluationA

Queue autouser runs against an evaluation. SPENDS Gemini tokens (real money) — once queued, cost is incurred. Each rating costs ~$0.091 (URL stimulus, SSE) to ~$0.137 (URL stimulus, SxS); one rating = one autouser × one comparison. With dryRun:true this tool is PREVIEW ONLY — NO runs are queued, NO cost is incurred. The dryRun response carries dryRun:true, queued:false, costEstimate, and wouldRun; the caller MUST re-issue this tool WITHOUT dryRun to actually queue the runs. Do not describe a dryRun response as 'kicked off', 'queued', or 'running'. Example: { evaluationId, autouserIds: ['au_1','au_2'] }.

ParametersJSON Schema
NameRequiredDescriptionDefault
evaluationIdYesEvaluation id to run autousers on.
autouserIdsYesOrdered list of autouser ids; duplicates produce distinct runs.
comparisonIdsNoOptional: subset of comparison ids on this eval; omit to fan out to all.
replaceRunIdNoOptional: terminal-state run id to replace in-place (used by retry).
dryRunNoPREVIEW ONLY. When true, NO autouser runs are queued and NO Gemini cost is incurred. Returns `costEstimate` and `wouldRun` so the caller can confirm spend before committing. The caller MUST re-issue this tool WITHOUT dryRun to actually queue the runs. Do NOT report runs as 'queued', 'started', 'kicked off', or 'running' on a dryRun response — by definition nothing was queued.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description fully discloses the tool's side effects: it spends real money, queues runs, and provides cost estimates. It details the behavior of dryRun vs real execution and warns against misreporting dryRun responses. This goes well beyond the annotations (which only indicate open world) and adds critical behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear primary action, followed by details, warnings, and an example. Every sentence adds value, though it is slightly verbose. It efficiently conveys complex behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (queueing, cost, dry run, multiple parameters) and the presence of an output schema, the description covers all essential aspects: cost, dry run process, response contents, and correct terminology. It leaves no critical gaps for an AI agent to misuse the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage, so the baseline is 3. The description provides an example and reiterates dryRun semantics, but does not add significant new meaning beyond what the schema already provides for each parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool queues autouser runs against an evaluation, explicitly distinguishing between dry runs and actual queueing. It includes specific cost details and an example, and its purpose is distinct from sibling tools like autousers_run_stop.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides strong guidance on using the dryRun parameter for preview and mandates re-issuing without dryRun to commit. It explains the cost implications and the response format. However, it does not explicitly mention alternatives or when not to use this tool relative to siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_run_stopStop autouser runs on an evaluationA
Idempotent

Cancel pending/running runs. Example: { evaluationId } (omit runIds to cancel all).

ParametersJSON Schema
NameRequiredDescriptionDefault
evaluationIdYesEvaluation id.
runIdsNoOptional: subset of run ids to cancel; omit to cancel all active runs.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already provide hints (destructiveHint false, idempotentHint true). The description adds the nuance of partial cancellation but does not disclose further behavioral traits such as impact on active runs or reversibility. No contradiction with annotations is evident.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: a single sentence with an example. Every word is meaningful, and it is front-loaded with the core action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema and full parameter coverage, the description is complete enough for typical usage. However, it could briefly mention any side effects or conditions (e.g., what happens to runs already in progress), but this is not critical.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the description is not required to add much. The example illustrates usage of evaluationId and the optional runIds, but adds minimal semantic value beyond the schema definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool cancels pending/running runs, with a concrete example. Among siblings like 'autousers_run' which starts runs, this tool's purpose is distinctly identified.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides an example and notes that omitting runIds cancels all runs, which is helpful usage guidance. However, it does not explicitly state when not to use this tool or mention alternatives, though the context is clear from sibling names.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autouser_status_getGet autouser run status snapshotA
Read-onlyIdempotent

Return run statuses + summary counts for an evaluation. Example: { evaluationId: 'ev_123' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
evaluationIdYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, and idempotentHint=true. The description adds 'snapshot' implying a point-in-time view, but no further behavioral context (e.g., what if evaluationId is invalid).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: one for purpose and one with an example. No filler, front-loaded, and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only tool with one parameter, existing annotations, and an output schema, the description is complete enough. It explains the tool's function and includes a concrete example.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear parameter description. The description provides an example but no additional meaning beyond the schema. Baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Return run statuses + summary counts' for an evaluation, specifying the exact resource and action. This distinguishes it from sibling tools like 'autouser_run_get' (which likely gets a single run) and 'autousers_run' (which initiates a run).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives, nor any exclusions. However, the tool is obviously read-only and snapshots, so it's implicitly for checking status without modification.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

autousers_updateUpdate an autouserB

PATCH a custom autouser; only supplied fields change. Example: { id, systemPrompt: 'Updated...' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAutouser id (URL path param).
nameNo
descriptionNo
roleNo
avatarNo
systemPromptNo
statusNo
visibilityNo
capabilitiesNoPersona metadata bundle (stored in `capabilities` JSON column).
configNoRuntime model config (stored in `config` JSON column).

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
roleNo
descriptionNo
statusNo
visibilityNo
isSystemNo
sourceNo
capabilitiesNo
configNo
createdAtNo
updatedAtNo

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations (readOnlyHint=false, destructiveHint=false) indicate a write operation that is not destructive. The description adds 'only supplied fields change', confirming partial updates and non-destructive behavior. However, it does not elaborate on authorization, limits, or side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is one sentence with an example, achieving brevity. The example adds value by showing a typical usage. However, it could be slightly more informative without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (10 parameters, nested objects) and low schema coverage, the description is insufficient. It does not mention the nested structure of capabilities and config, nor does it clarify constraints or when to use sub-fields. The example is too minimal.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is low (30%), and the description only provides an example with id and systemPrompt. It does not explain the 8 other parameters, including nested objects like capabilities and config, leaving the agent to infer from the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it performs a PATCH update on a custom autouser, modifying only supplied fields. It distinguishes itself from siblings such as autousers_create, autousers_delete, and autousers_get by the verb and partial update behavior.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not provide guidance on when to use this tool versus alternatives like autousers_create or autousers_duplicate. There is no context about prerequisites, scope, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_agreement_getGet inter-rater agreement (Cohen's Kappa)A
Read-onlyIdempotent

Pairwise Cohen's Kappa across raters with cache-version-aware results. Returns null/insufficient when <2 raters overlap. Example: { id }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, non-destructive, idempotent behavior. The description adds context about cache-version-awareness and return null when fewer than 2 raters overlap, going beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences and a compact example provide essential information with no redundancy or fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read tool with one required parameter and an output schema, the description covers core behavior, edge cases, and caching, making it fully adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with the parameter description 'Evaluation id.' The example '{ id }' adds little extra meaning; baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it computes pairwise Cohen's Kappa for inter-rater agreement, distinguishing it from other evaluation tools like evaluations_results_get or evaluations_ratings_list.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for agreement analysis but does not explicitly mention when to use this tool over siblings or provide exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_ai_insights_getGet AI-generated evaluation insightsA
Read-onlyIdempotent

Gemini-authored summary + key findings + recommendations from aggregate scores and per-dimension averages. 503 when Gemini is unconfigured. Example: { id }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond annotations (readOnlyHint, idempotentHint), the description adds the critical error condition '503 when Gemini is unconfigured,' which is valuable for the agent. It also describes the output content, though an output schema exists. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short and front-loaded, but the fragment 'Example: { id }' adds little value and could be integrated better. Overall efficient, but not maximally concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (1 param, high schema coverage, annotations, output schema), the description covers the main aspects: function, error case, and input example. Could briefly mention that the evaluation must exist, but 4 is fair.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with the parameter 'id' described as 'Evaluation id.' The description adds no extra semantic detail (the 'Example: { id }' is trivial). Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it returns a 'Gemini-authored summary + key findings + recommendations from aggregate scores and per-dimension averages,' which is specific and distinct from siblings like evaluations_get or evaluations_results_get. The error case (503) further clarifies purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use when AI-generated insights are needed, but does not explicitly instruct when to use this tool versus alternatives such as evaluations_results_get for raw data. No when-not or alternative tool names are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_createCreate an evaluationA

Create a new evaluation (SSE or SxS) and persist it to the database.

DEFAULT POLICY — preview, then confirm:

  1. First call with dryRun: true to show the user the proposed config + cost estimate + the personas you'd suggest. Do NOT describe a dryRun response as 'created', 'live', or 'started' — by definition nothing was persisted.

  2. Wait for the user to confirm or adjust (dimensions, autousers, instructions, scenario).

  3. Re-issue WITHOUT dryRun, almost always with status='Draft'. Drafts are iteratively editable via evaluations_update.

  4. Only set status='Running' when the user explicitly says to launch and the eval is fully configured (including selectedAutousers if evaluationMethod is 'ai'/'both').

When status='Running' AND selectedAutousers is non-empty AND evaluationMethod is 'ai'/'both', this ALSO queues autouser runs and returns a cost estimate.

Autousers: NEVER pick personas yourself; they're contextual. See selectedAutousers field docs for the right workflow (autousers_list first, custom personas before built-ins, propose then confirm). If the user wants only human raters, use evaluationMethod: 'manual' and omit selectedAutousers — that's a fully valid eval, not a misconfiguration.

Dimensions: selectedDimensionIds defaults to ['overall'] when omitted/empty; prefer built-in ids ('overall', 'usability', 'visual-design', 'accessibility', 'content', 'helpfulness', 'accuracy', 'safety', 'design-system') or ids from templates_list. Any id in customDimensions[] MUST also appear in selectedDimensionIds, and each customDimension MUST have an id and a name (or label).

Example (preview): { dryRun: true, name: 'Homepage redesign', type: 'SxS', comparisonPairs: [{id:'p1', currentUrl:'https://a.com', variantUrl:'https://b.com'}], selectedDimensionIds: ['overall', 'usability'], evaluationMethod: 'ai', selectedAutousers: [{autouserId:'novice', agentCount:1}] }.

ParametersJSON Schema
NameRequiredDescriptionDefault
teamIdNo
nameYesHuman-readable evaluation name.
descriptionNo
typeNoSSE = single-stimulus eval, SxS = side-by-side.
statusNoSTRONGLY PREFER omitting this field — the server defaults to 'Draft' and that is the right choice for ~99% of evals. A Draft eval can be iteratively refined: add/remove dimensions, swap autousers, edit instructions, all without delete-and-recreate. Once the user is satisfied, flip status to 'Running' via evaluations_update — that promotes the eval AND auto-promotes any draft custom dimensions to active. Setting status:'Running' on creation locks the eval into the published state immediately and forces destructive recreate for any iteration. Only pass 'Running' when the user explicitly says 'publish now' / 'launch immediately' / 'start running it' with no expectation of further edits.
shareAccessNo
sharePasswordNo
shareRequireEmailNo
shareRequireNameNo
shareAllowAnonNo
shareExpiryNoISO 8601 timestamp; share link expires after this.
allowMultipleRatingsNo
designUrlsNoSSE only: list of design stimuli. Each item: {id, url?, label?, stimulusType?, fileId?, fileUrl?}.
comparisonPairsNoSxS only: list of A/B pairs. Each item: {id, currentUrl?, variantUrl?, label?, sideAType?, sideBType?, ...}.
selectedDimensionIdsNoDimension ids the rating UI scores on. Defaults to ['overall'] when omitted or empty. Prefer built-in ids: 'overall', 'usability', 'visual-design', 'accessibility', 'content', 'helpfulness', 'accuracy', 'safety', 'design-system'. For team-specific dimensions, call templates_list first and reuse the ids it returns. Every id in customDimensions[] MUST also appear here — the API rejects mismatches.
customDimensionsNoFree-form custom dimensions for this eval. Each item MUST include `id` plus a display `name` (or `label`); every id here MUST also appear in selectedDimensionIds. WHEN TO USE THIS vs templates_create: - DEFAULT path: inline customDimensions here. The server auto-saves each dimension as a team-scoped template row in the same transaction as the eval. Dimensions appear on /templates as soon as the eval transitions to Running (status='Running' on create, OR a later evaluations_update that flips status to 'Running'). Until then they're saved as drafts (visible on /templates with the Drafts filter). Use this for ANY flow where the user is creating an eval — explicit, draft, ad-hoc, reusable, all of them. Do NOT call `templates_create` separately for each dimension before this — it will create duplicate template rows. - Use `templates_create` ONLY when the user explicitly says they want a STANDALONE template with NO eval yet. For example: 'create a template for design-system compliance and save it' / 'add this dimension to my library, I'll use it later' with no mention of running an eval. That endpoint creates the dim row directly without an eval reference. ALWAYS include 4 contributing factors per dimension (3 minimum, 5 absolute max — only exceed 4 if the user explicitly insists, decision fatigue degrades rating quality past that). For SxS evals supply `factors[]`; for SSE supply `sseCriteria[]` (or both if you don't know which the eval will use). Each factor/criterion shape: `{id, label, description}` where `id` is a stable snake-case slug like `search-relevance-noise`, `label` is 1–3 words shown as the checkbox text (e.g. 'Signal vs noise'), and `description` is one sentence explaining what the rater is looking for. Match the style of built-in dimensions: factors are the *contributors* to the rating, not synonyms of the dimension itself. Example for a custom 'Search relevance' dimension on an SxS eval: factors: [ {id:'search-relevance-match', label:'Intent match', description:'Results align with what the user actually wants to buy.'}, {id:'search-relevance-noise', label:'Signal vs noise', description:'Filters out irrelevant or distracting results.'}, {id:'search-relevance-coverage', label:'Coverage', description:'Captures the relevant breadth without obvious gaps.'}, {id:'search-relevance-confidence', label:'Confidence', description:'Helps the rater feel confident the answer is correct.'}, ] If you skip factors/sseCriteria, the server fills 4 generic-but-honest fallbacks derived from the dimension name + description — but those are markedly less useful than dimension-specific ones, so always supply real ones when you have enough context.
selectedAutousersNoAutousers (AI personas) to run, plus per-autouser agentCount. The MCP NEVER picks personas for you — selection is contextual to what the user is evaluating, and generic defaults pollute the team's library and misrepresent the eval. Workflow: 1. **Decide the rating method first.** If the user only wants human raters (manual UX testing, sharing with a team, etc.), set `evaluationMethod: 'manual'` and OMIT this field. No autousers will be attached and that's the right outcome — don't pretend the eval needs them. 2. **For AI or hybrid evals**, propose 1–4 personas tailored to the eval's domain. ALWAYS call `autousers_list` first to surface the team's custom personas — domain-specific autousers (e.g. a 'Tokyo bilingual commuter' the team already calibrated) almost always beat generic built-ins. Then fall back to built-ins (`novice`, `power-user`, `mobile-user`, `design-critic`, `keyboard-navigator`, `senior-user`, `slow-network`, `screen-reader`) only as needed to round out perspectives. 3. **Preview-then-confirm.** Use `dryRun: true` first to show the user the proposed config + cost estimate + persona picks, then re-issue without dryRun once they confirm. Don't ship to Running on the first call — leave status='Draft' so the user can iterate. Validation: when `evaluationMethod` is 'ai'/'both' and this field is empty, the response includes a `warnings: [{ code: 'ai_eval_without_autousers' }]` entry — address it on the next turn (attach personas or switch to manual). Trying to set `status: 'Running'` in that state hard-fails.
evaluationMethodNo
instructionsNo
scenarioNo
skipPreQualificationNo
hideSliderNo
hideOpenTextQuestionsNo
defaultLayoutNo
ratingFlowNoDO NOT pass this field unless the user explicitly requests 'step-by-step' or 'one question at a time' wording — the server defaults to 'combined' and that is the right choice for ~99% of evals. Combined renders the rating slider, contributing factors, and open-text inline on a single screen per dimension; stepped fragments those into 3 sub-steps (rating → factors → open-text) and is only useful when each dimension has 5+ open-text questions to amortize the extra clicks. Passing 'stepped' on a normal-sized eval makes the rater click through empty intermediate screens — actively bad UX.
dryRunNoPREVIEW ONLY. When true, NO database write occurs and NO evaluation is created — the response is a config preview plus a cost estimate. The caller MUST re-issue this tool WITHOUT dryRun to actually persist the evaluation. Do NOT report the eval as 'created', 'live', 'started', or 'queued' on a dryRun response — by definition nothing was persisted.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description thoroughly explains behavioral traits beyond annotations, such as dryRun not persisting, the effect of setting status='Running' (locks eval, auto-promotes custom dimensions), and the default behavior of ratingFlow. No contradictions with annotations (readOnlyHint=false, destructiveHint=false).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is lengthy but well-organized with clear sections and bullet points. It front-loads the default policy and key guidelines. A few phrases are repeated (e.g., dryRun warning appears in multiple places), but overall it's efficient for the complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 26 parameters, 1 required, and the presence of an output schema, the description covers all critical aspects: creation workflow, preview-confirm pattern, status lifecycle, dimension management, autouser selection, and explicit references to sibling tools. No gaps remain for the agent to misuse.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With only 42% schema description coverage, the description adds substantial meaning to parameters like status (defaults, iteration vs. final), customDimensions (when to use inline vs. templates_create), selectedAutousers (workflow, fallback), and dryRun (preview only). It explains schema fields that have minimal descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states 'Create a new evaluation (SSE or SxS) and persist it to the database.' It also distinguishes from sibling tools like evaluations_update and templates_create, and clarifies the creation workflow vs. editing. The purpose is specific and clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides detailed guidance on when to use dryRun vs. real creation, when to set status fields, and when to use templates_create vs. inline customDimensions. It explicitly defines the preview-then-confirm workflow and warns against common mistakes (e.g., never picking personas yourself).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_deleteDelete an evaluationA
DestructiveIdempotent

Hard-delete an evaluation and its dependent rows (cascades to comparisons/ratings). Admin+ required. Example: { id: 'eval_…' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
deletedYes
idYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond annotations (destructiveHint, idempotentHint), the description adds critical details: hard-delete cascades to comparisons/ratings, and requires admin permissions. This fully discloses behavioral impact.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences: action+scope, permission, example. No waste, front-loaded with key information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple single-parameter tool with full schema and annotations, the description covers all necessary aspects: operation, cascading effect, permissions, and example. Output schema exists, so return details are not needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema covers 100% of parameters with description; the description adds an example but no extra semantic nuance beyond what the schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the action (hard-delete) and resource (evaluation and its dependent rows), distinguishing it from sibling tools like evaluations_list or evaluations_create.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states permission requirement ('Admin+ required') and provides an example call. However, no explicit when-not or alternative suggestions are given, though the context makes it clear this is the deletion tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_export_getExport evaluation resultsA
Read-onlyIdempotent

Download an eval's results as JSON (default) or CSV (one row per rating × dimension). Example: { id, format: 'csv' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.
formatNoDefaults to 'json'. CSV emits one row per rating × dimension.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnlyHint=true, destructiveHint=false, idempotentHint=true, which align with the 'Download' verb. The description adds behavioral detail: 'CSV emits one row per rating × dimension', providing context beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise at two sentences, front-loads the purpose, and includes a concrete example. Every word is necessary, with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (export with two parameters), the combination of annotations, schema, and description provides complete guidance. The output schema exists (though not shown), and the description covers the key behavioral aspects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for both parameters. The description adds value by explaining the default for 'format' and the row structure for CSV, which is not in the schema. This exceeds the baseline of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Download an eval's results as JSON (default) or CSV', specifying the verb ('download'), resource ('eval's results'), and output formats. This distinguishes it from sibling tools like evaluations_get or evaluations_results_get.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides an explicit example ({ id, format: 'csv' }) and mentions defaults, giving clear context for usage. While it doesn't explicitly state when not to use it, the tool's purpose is straightforward and the example suffices.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_getGet an evaluation by IDA
Read-onlyIdempotent

Fetch a single evaluation by id (includes config, comparisons, autouser selections). Example: id from evaluations_list.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
nameNo
typeNo
statusNo
teamIdNo
createdAtNo
updatedAtNo
ratingsCountNo
comparisonsCountNo
autouserRunSummaryNo
isSharedWithMeNo
myPermissionNo
linksNo

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false. The description adds value by specifying what data is included (config, comparisons, autouser selections). It does not contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence plus an example, front-loaded and efficient. Every word serves a purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple input schema (one parameter) and presence of an output schema, the description covers the essential purpose, included data, and an example. No gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with the single 'id' parameter. The description adds a hint about the source ('id from evaluations_list'), but this is minimal extra value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Fetch a single evaluation by id' with specific verb and resource, and mentions included data (config, comparisons, autouser selections). This distinguishes it from siblings like evaluations_list (listing) and evaluations_delete.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by saying 'Example: id from evaluations_list.' but does not explicitly state when to use this tool over alternatives or provide exclusions. For a simple tool with many siblings, more explicit guidance would help.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_listList evaluationsA
Read-onlyIdempotent

List evaluations the caller can see. Example: 'show me my running evals' — call with no args.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMax items to return (1-100, default 20).
starting_afterNoCursor: id of the last item from the previous page.
teamIdNoOptional: scope to a specific team id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
dataYes
has_moreNo
next_cursorNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true, so the description correctly aligns with these. The description adds that it lists evaluations the caller can see, clarifying the access scope. No contradictions or missing critical behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: one sentence plus an example. Every word is meaningful, no redundancy. It is front-loaded with the core function and immediately followed by a practical example.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple with no required parameters and an output schema provided. The description and schema together cover the essential usage. However, it could briefly mention pagination behavior (e.g., cursor usage) to be fully self-contained, but it is still adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage for all three parameters (limit, starting_after, teamId) with clear details. The description does not add any additional meaning beyond what the schema already provides, so the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'list' and the resource 'evaluations the caller can see', which is specific and unambiguous. It distinguishes from sibling tools like evaluations_get or evaluations_create by its listing nature and scope of visible items.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides an example of a typical query ('show me my running evals') and notes it can be called with no args, implying common usage. However, it does not explicitly state when to use this tool versus alternatives like evaluations_get or evaluations_results_get, nor does it mention any prerequisites or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_ratings_listList ratings for an evaluationA
Read-onlyIdempotent

List ratings (human + autouser) for one evaluation. Example: evaluationId from evaluations_list.

ParametersJSON Schema
NameRequiredDescriptionDefault
evaluationIdYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true. Description adds that ratings include human and autouser, consistent with annotations but not significantly extending beyond them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two-sentence description with no wasted words; front-loaded with action and resource.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Low complexity (one parameter, output schema present). Description is sufficient for a list operation, though lacks details on pagination or ordering—minor gap given output schema likely covers structure.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter, which is described as 'Evaluation id.' Description adds an example of where to get the ID, providing practical context without new semantic information.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear verb+resource: 'List ratings' for an evaluation, specifying inclusion of human and autouser ratings. Distinguishes from sibling tools like evaluations_list and evaluations_results_get.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Only hints at getting evaluationId from evaluations_list but provides no guidance on when to use this tool versus alternatives such as evaluations_results_get or autouser_run_turns_list.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_results_getGet evaluation resultsA
Read-onlyIdempotent

Aggregate stats + per-rater summaries + pairwise Kappa agreement (when 2 raters overlap). Example: { id }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true. Description adds that tool returns aggregate stats, per-rater summaries, and pairwise Kappa agreement, providing behavioral context beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is one sentence plus a short example, containing only essential information with zero waste. Front-loads key outputs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has one parameter, clear annotations, and an output schema, the description adequately explains what is returned. Could mention if paginated or not, but overall sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter 'id' with description 'Evaluation id.' Description adds an example usage '{ id }' but adds no additional semantic meaning beyond schema. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns 'Aggregate stats + per-rater summaries + pairwise Kappa agreement', specifying verb (get) and resource (evaluation results). It distinguishes from siblings like evaluations_get or evaluations_list by detailing the specific outputs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for detailed results but does not explicitly state when to use this tool versus alternatives like evaluations_get or evaluations_ratings_list. No exclusions or context given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_save_draftSave evaluation draft (wizard progress)A
Idempotent

Merge wizard fields into a Draft/Ended eval's metadata (regenerates Comparison rows when designUrls/comparisonPairs change). Example: { id, designUrls: [...] }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id (path param).
nameNo
descriptionNo
typeNo
designUrlsNoSSE only — replaces the draft's design list.
comparisonPairsNoSxS only — replaces the draft's A/B pairs.
selectedDimensionIdsNo
customDimensionsNo
selectedAutousersNo
evaluationMethodNo
instructionsNo
scenarioNo
templateIdNo
customQuestionsNo
selectedPersonasNo
agentCountNo
sideAUrlNo
sideBUrlNo
skipPreQualificationNo
hideSliderNo
hideOpenTextQuestionsNo
defaultLayoutNo
ratingFlowNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
nameNo
typeNo
statusNo
teamIdNo
createdAtNo
updatedAtNo
ratingsCountNo
comparisonsCountNo
autouserRunSummaryNo
isSharedWithMeNo
myPermissionNo
linksNo

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare idempotentHint=true, and the description adds the behavioral detail of regenerating comparison rows. This is consistent and provides useful context beyond annotations. However, it lacks disclosure on authorization or other state effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences plus an example are highly concise and front-loaded. Every piece of information earns its place without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite high complexity (23 parameters), the description covers only the core merge and one side effect. It omits details about other fields, partial updates, or status constraints, leaving the agent under-informed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With only 13% schema coverage, the description barely adds meaning beyond the schema. Only 'id' and 'designUrls' are mentioned via the example; the other 21 parameters are left unaddressed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Merge wizard fields into a Draft/Ended eval's metadata') and a key side effect ('regenerates Comparison rows when designUrls/comparisonPairs change'), with an example. This is specific and distinguishes it from sibling tools like evaluations_update.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The title mentions 'wizard progress' implying usage context, but the description does not explicitly state when to use this tool versus alternatives like evaluations_update. No when-not or alternative guidance is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_share_createGrant a per-user share on an evaluationA

Upsert an EvaluationShare granting a user VIEWER/EDITOR/OWNER access. Admin+ required. Example: { id, userId, permission: 'VIEWER' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id (path param).
userIdYesTarget user's CUID — must be an existing User.
permissionYesRole to grant on this evaluation.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Description aligns with annotations (mutation, not read-only/destructive). Adds that admin permission is needed and illustrates an example. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences, front-loaded with key information (action, resource, example). No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With output schema defining return values, description adequately covers upsert behavior, permission roles, and auth requirement. Missing edge cases like duplicate shares, but acceptable.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% coverage with clear descriptions. Description adds a concrete example but does not significantly enhance meaning beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Title and description clearly state the action (Upsert/grant share) and resource (evaluation share). It distinguishes from siblings like evaluations_shares_list and evaluations_transfer.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States 'Admin+ required' and provides an example, giving clear context for when to use. Does not explicitly mention when not to use or alternatives, but no conflicting siblings exist.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_shares_listList per-user shares on an evaluationA
Read-onlyIdempotent

List explicit EvaluationShare rows (VIEWER/EDITOR/OWNER) for one eval. Admin+ required. Example: { id }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true. The description adds the behavioral requirement of 'Admin+ required' and clarifies that it lists explicit shares. This adds value beyond annotations without contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no redundant information. It efficiently states purpose, requirement, and an example. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with one parameter and an output schema, the description adequately covers the purpose, permission requirement, and gives an example. It could mention that it lists only explicit shares (no inherited), but overall complete enough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents the single parameter 'id' with a description. The description adds a minimal example but no additional semantic meaning beyond what the schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'list', the resource 'EvaluationShare rows', and specifies the scope 'for one eval' with the share types (VIEWER/EDITOR/OWNER). It distinguishes from sibling tools like evaluations_share_create.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions 'Admin+ required' as a usage context but does not provide explicit when-to-use or when-not-to-use guidance relative to alternative tools for managing shares. It is clear but lacks exclusion criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_transferTransfer evaluation ownership to another userA

Promote a user to OWNER share on this eval and demote the caller to EDITOR. Admin+ required. Example: { id, userId }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id (path param).
userIdYesUser id to promote to OWNER. Must not equal the caller.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
nameNo
typeNo
statusNo
teamIdNo
createdAtNo
updatedAtNo
ratingsCountNo
comparisonsCountNo
autouserRunSummaryNo
isSharedWithMeNo
myPermissionNo
linksNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate the tool is not read-only or idempotent, and description adds that the caller is demoted to EDITOR and requires Admin+. This provides useful behavioral context beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with an example, no redundant information. Every word serves a purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's straightforward nature (transfer ownership), the description covers purpose, permissions, and parameter usage. Output schema exists so return values need not be described.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers both parameters with descriptions (100% coverage). Description adds value by noting Admin+ requirement and the example, but does not add syntactic details beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The title and description clearly state the tool transfers evaluation ownership to another user, specifying the caller is demoted to EDITOR and the target becomes OWNER. This distinguishes it from sibling tools like evaluations_share_create.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly requires Admin+ and provides an example, giving clear context for usage. However, it does not explicitly contrast with alternative tools like evaluations_share_create.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluations_updateUpdate an evaluationA

Patch any subset of an eval's fields. This is the single tool to use for editing an existing eval — pass scalar fields (name/description/status/share*), wizard fields (selectedAutousers, selectedDimensionIds, customDimensions, designUrls, comparisonPairs, instructions, scenario, evaluationMethod, defaultLayout, ratingFlow, etc.), or a mix. The MCP partitions the input internally and calls both PATCH /evaluations/[id] and PATCH /evaluations/[id]/draft as needed; you do NOT need to call evaluations_save_draft separately.

Common pattern after evaluations_create returns a Draft with warnings: [{ code: 'ai_eval_without_autousers' }]: re-issue this tool with the chosen autousers, e.g. { id, selectedAutousers: [{autouserId:'novice', agentCount:1}, ...] }. To then publish: { id, status: 'Running' }. To do both at once: { id, selectedAutousers: [...], status: 'Running' }.

Auth: Editor+ for most fields; Admin+ for type/teamId/shareAccess/sharePassword.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesEvaluation id (path param).
teamIdNoAdmin+ only — move the eval to a different team.
nameNo
descriptionNo
typeNoAdmin+ only.
statusNo
shareAccessNoAdmin+ only.
sharePasswordNoAdmin+ only. Min 4 chars when shareAccess=PASSWORD_PROTECTED.
shareRequireEmailNo
shareRequireNameNo
shareAllowAnonNo
shareExpiryNoISO 8601 timestamp; pass empty string or omit to clear.
allowMultipleRatingsNo
browserEngineNonull clears the override → inherits user default.
useProxyNonull clears the override → inherits user default.
proxyRegionNoProxy region id; null clears the override.
modelIdNoGemini model id; null clears the override.
selectedAutousersNoAutousers to attach (replaces the existing list). Pass an array of {autouserId, agentCount}. Internally routed to /draft. To remove all autousers, pass [].
selectedDimensionIdsNoDimension ids the rating UI scores on (replaces the existing list). Internally routed to /draft.
customDimensionsNoCustom dimensions for this eval (replaces the existing list). Each id MUST also appear in selectedDimensionIds. Internally routed to /draft.
designUrlsNoSSE only — replaces the design list. Internally routed to /draft.
comparisonPairsNoSxS only — replaces the A/B pairs. Internally routed to /draft.
evaluationMethodNoInternally routed to /draft.
instructionsNoInternally routed to /draft.
scenarioNoInternally routed to /draft.
skipPreQualificationNoInternally routed to /draft.
hideSliderNoInternally routed to /draft.
hideOpenTextQuestionsNoInternally routed to /draft.
defaultLayoutNoInternally routed to /draft.
ratingFlowNoInternally routed to /draft.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
nameNo
typeNo
statusNo
teamIdNo
createdAtNo
updatedAtNo
ratingsCountNo
comparisonsCountNo
autouserRunSummaryNo
isSharedWithMeNo
myPermissionNo
linksNo

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds behavioral context beyond annotations: it reveals that the tool internally partitions input and calls both PATCH endpoints, so agents understand the underlying mechanics. It also explains auth requirements. Annotations only provide readOnlyHint=false and destructiveHint=false, which are consistent. The description does not contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is moderately long but well-structured: first sentence states purpose, then common pattern with example, then auth note. It avoids fluff and front-loads key information. Given the complexity of the tool (30 parameters), the length is justified.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers purpose, usage pattern, auth, and internal routing. The output schema exists, so return values are not needed. For a tool with 30 parameters, the description provides sufficient context for an agent to know when and how to use it, and what to expect behaviorally.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 77%, so the schema already documents most parameters. The description reiterates that selectedAutousers, selectedDimensionIds, etc., are routed to /draft, but this is also in the schema descriptions. The description adds a common pattern example, which is helpful but not extensive. Given the high coverage, the added value is marginal.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that this is the single tool for editing an existing evaluation, patching any subset of fields. It distinguishes itself from siblings like evaluations_save_draft by noting that the MCP internally handles both endpoints, so the agent knows not to call that separate tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: when to use (editing an existing eval), a common pattern after evaluations_create with warnings, how to publish by setting status, and the ability to update autousers and publish in one call. It also mentions auth requirements (Editor+ for most, Admin+ for type/teamId/shareAccess). This leaves no ambiguity about context or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

team_members_listList team membersA
Read-onlyIdempotent

List members of a team (id, role, user profile). Example: { teamId: 'team_abc123' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
teamIdYesTeam id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, idempotentHint=true, so the description's burden is low. The description adds no extra behavioral context (e.g., pagination, ordering, membership scope). Still, the annotations cover the safety profile adequately.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence with an inline example, no wasted words. It is front-loaded and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema (not shown) which covers return values. The description is adequate for a straightforward list operation, but could mention behavior like ordering or whether it lists all members or only direct ones. Given the simplicity, a score of 3 is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (teamId described as 'Team id.'). The description adds an example value which provides a concrete format but does not add new semantic meaning beyond the schema. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'List' and the resource 'members of a team', listing the returned fields (id, role, user profile). However, it does not differentiate from sibling tools like teams_list or teams_get, though no sibling specifically lists members, so it remains clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a concrete usage example ({ teamId: 'team_abc123' }) which guides the agent on how to invoke the tool. No explicit exclusions or when-not-to-use guidance is given, but given the absence of conflicting siblings, the example suffices.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

teams_getGet a teamA
Read-onlyIdempotent

Fetch one team's detail including the full member list. Example: { id: 'team_abc123' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesTeam id.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnly, non-destructive, idempotent. Description adds that it includes the full member list, providing extra behavioral context beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two efficient sentences: first states purpose and scope, second gives an example. No fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple get tool with an output schema, the description covers purpose, parameter example, and result scope (full member list). Complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% coverage with description for 'id'; description provides an example format, adding value beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states 'Fetch one team's detail including the full member list' with a specific verb and resource, distinguishing it from sibling tools like 'teams_list' and 'team_members_list'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use vs. alternatives; although the purpose implies it is for fetching a single team detail, it could benefit from noting that 'teams_list' is for listing without full details.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

teams_listList teamsA
Read-onlyIdempotent

List every team the caller belongs to (id, name, role, member count). Example: {} — no params.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
dataYes
has_moreNo
next_cursorNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, destructiveHint, and idempotentHint. The description adds behavioral context about scope (teams the caller belongs to) and return fields, which complements the annotations well.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: one sentence and an example. Every part is essential, with no waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite its simplicity, the description covers all necessary aspects: purpose, scope, return fields, and parameter expectations. An output schema exists for further detail, making this complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no parameters, and the description confirms this with 'no params'. Since schema coverage is 100%, the description adds minimal value but is accurate and concise.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb (List), resource (every team the caller belongs to), and includes return fields (id, name, role, member count). It distinguishes from sibling tools like teams_get (single team) and team_members_list (members of a team) by specifying scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage through 'no params' and an example, but lacks explicit guidance on when to use this tool vs alternatives like teams_get. However, the simplicity of the tool reduces the need for such guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

templates_createCreate a templateA

Create and persist a team-scoped template (rubric of dimensions). REQUIRES teamId and name — call teams_list first if you do not know the teamId. The template is persisted only if this tool returns a success response (no isError) AND the response object contains a non-empty id field. Verify both before reporting success to the user. If the response carries isError: true OR is missing an id, the template was NOT created — surface the error message verbatim instead of claiming success. Example: { teamId: 'cmoi…', name: 'UX heuristics', type: 'TEXT_SSE' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
teamIdYesREQUIRED. Team id (cuid) that will own the template. Must be a team the caller is an Editor+ on. If you do not know the teamId, call teams_list first; do NOT guess or invent one — an invalid teamId returns a 400/404 and the template is NOT persisted.
nameYesHuman-readable template name.
descriptionNo
typeNoStimulus + mode (default TEXT_SXS).
scaleTypeNo
scaleMinNo
scaleMaxNo
scaleLabelsNoMap of score value (string) → label, e.g. {'1':'Bad'}.
guidelinesNo
criteriaNo
iconNo
scaleQuestionNo
sseQuestionNo
factorsNo
sseCriteriaNo
sseScaleLabelsNo
sxsScaleLabelsNo
sseScaleDescriptionsNo
sxsScaleDescriptionsNo
contextsNo
scoringModeNo
sseAnchorsNo
sseAnchorLabelsNo
sseAnchorDescriptionsNo
sxsScoringModeNo
sxsAnchorsNo
sxsAnchorLabelsNo
sxsAnchorDescriptionsNo
isPrimaryNo
sourceBuiltInIdNo
openTextEnabledNo
customQuestionsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
descriptionNo
typeNo
scaleTypeNo
isSystemNo
createdAtNo
updatedAtNo

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate the tool is not read-only (readOnlyHint=false), not destructive (destructiveHint=false), and not idempotent. The description adds that persistence only occurs on specific success conditions, and that an invalid teamId leads to 400/404 errors. This provides useful behavioral context beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is clear and front-loaded with the core purpose. It includes important usage and success verification details without being excessively long. The example adds value. Slightly verbose but efficient overall.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (32 parameters, nested objects), the description covers critical success criteria and prerequisite inputs. However, it lacks details on how other parameters affect behavior or the output schema. It is adequate for basic creation but not fully comprehensive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 13% (many parameters lack descriptions). The description adds meaning for teamId and name (required, prerequisite for teamId), and provides an example. However, it does not explain the purpose or usage of the other 30 parameters, which is a significant gap given the low schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool creates a team-scoped template, distinguishing it from siblings like templates_delete or templates_get. It specifies the resource (template) and action (create), and gives required parameters teamId and name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says to call teams_list if teamId is unknown, and provides detailed success verification conditions (no isError and non-empty id). Also warns about invalid teamId causing errors. This guides the agent on when and how to use the tool correctly.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

templates_deleteDelete a templateA
DestructiveIdempotent

Hard-delete a team-scoped template. Refuses if attached to an evaluation. Example: { id: 'tpl_abc' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesTemplate id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
deletedYes
idYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses destructive action ('Hard-delete') and the refusal condition, adding context beyond annotations which already indicate destructive and idempotent traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences with front-loaded purpose, no extraneous information, and an illustrative example.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers purpose, behavioral constraints, and parameter example; output schema presumably handles return values, so no gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Provides an example usage with the required id parameter, adding concrete value beyond the schema description which only says 'Template id.'

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the tool hard-deletes a team-scoped template and refuses if attached to an evaluation, distinguishing it from siblings like templates_create or templates_update.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Describes a key condition ('Refuses if attached to an evaluation'), guiding when not to use, but does not explicitly mention alternatives or when-to-use scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

templates_duplicateDuplicate a templateA

Deep-clone a template into a destination team. Example: { id: 'tpl_abc', teamId: 'team_xyz' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesSource template id to clone.
teamIdNoOptional destination team id; defaults to caller's active team.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
descriptionNo
typeNo
scaleTypeNo
isSystemNo
createdAtNo
updatedAtNo

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare non-readOnly and non-destructive. The description adds 'deep-clone' but no behavioral details beyond that. Output schema covers return info, so overall adequate but not enriched.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence plus example, no fluff. Extremely concise and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the main action and provides an example. Output schema exists so return values are handled. Could optionally mention that source template must exist, but not a critical gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with parameter descriptions. The example adds value by showing the format ('tpl_abc', 'team_xyz'), which clarifies usage beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it deep-clones a template into a destination team, using specific verb and resource. It distinguishes from sibling tools like templates_create which creates from scratch.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use or when-not-to-use guidance is provided. The example implies usage but does not specify prerequisites or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

templates_getGet a template by IDA
Read-onlyIdempotent

Fetch a single template by id. Example: id 'tpl_abc123' returns the template's full config.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesTemplate id.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
descriptionNo
typeNo
scaleTypeNo
isSystemNo
createdAtNo
updatedAtNo

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is clear. The description adds the example and mentions 'returns the template's full config', which provides some insight into the output but is not extensive. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is just two sentences, front-loading the action and providing a concrete example. Every word is informative, with no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple get operation with one parameter, annotations covering safety, and an output schema present, the description is fully sufficient. It communicates what the tool does and gives a practical example.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for the single parameter 'id' (described as 'Template id.'). The tool description adds an example format ('tpl_abc123'), which provides concrete semantics beyond the schema's generic description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Fetch' and identifies the resource as a single template by ID, clearly distinguishing it from sibling tools like templates_list (which lists all) and templates_create. The example further concretizes the purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not explicitly state when to use this tool compared to alternatives like templates_list or templates_get for other resources. No exclusion criteria or context for use is provided, leaving the agent without guidance for selecting among siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

templates_listList question templatesA
Read-onlyIdempotent

List evaluation templates available to the caller. Example: 'show me my templates' — call with no args.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMax items to return (1-100, default 20).
starting_afterNoCursor: id of the last item from the previous page.

Output Schema

ParametersJSON Schema
NameRequiredDescription
dataYes
has_moreNo
next_cursorNo

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, non-destructive, idempotent. Description adds the 'available to the caller' scope, which is insightful. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence plus an example. Every word adds value. No fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With full schema coverage, annotations, and an output schema, the description is complete. It explains the scope and optionality sufficiently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with clear descriptions for both parameters. The description adds that 'no args' is typical, but no additional semantic value beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states verb 'List', resource 'evaluation templates', and scope 'available to the caller'. Distinguishes from sibling template tools like create, delete, etc. with an explicit example.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives (e.g., templates_get for a single template, templates_list for all). Only an example usage is given, lacking context for exclusion.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

templates_updateUpdate a templateB

Patch a template by id; only included fields change. Example: { id: 'tpl_abc', name: 'Renamed' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesTemplate id.
nameNo
descriptionNo
typeNo
scaleTypeNo
scaleMinNo
scaleMaxNo
scaleLabelsNo
guidelinesNo
criteriaNo
iconNo
scaleQuestionNo
sseQuestionNo
factorsNo
sseCriteriaNo
sseScaleLabelsNo
sxsScaleLabelsNo
sseScaleDescriptionsNo
sxsScaleDescriptionsNo
contextsNo
scoringModeNo
sseAnchorsNo
sseAnchorLabelsNo
sseAnchorDescriptionsNo
sxsScoringModeNo
sxsAnchorsNo
sxsAnchorLabelsNo
sxsAnchorDescriptionsNo
isPrimaryNo
openTextEnabledNo
customQuestionsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
teamIdNo
nameNo
descriptionNo
typeNo
scaleTypeNo
isSystemNo
createdAtNo
updatedAtNo

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the patch behavior (only included fields change) and gives an example. Annotations show readOnlyHint=false (write), destructiveHint=false (not destructive). No contradictions. However, it lacks details on side effects, validation, or required permissions, which the minimal annotations don't cover.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very concise: one sentence plus an example. It's front-loaded with the action. However, it sacrifices completeness for brevity. Every element earns its place, but more detail could be added without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (31 params, nested objects), the description is too sparse. Output schema exists to explain return values, but the description does not cover validation rules, field dependencies, or typical usage patterns. It leaves the AI agent to infer most behavior from the schema alone.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema has 31 parameters but only 3% have descriptions (only 'id' is documented). The description does not explain any other parameters beyond the example using 'name'. It fails to compensate for the schema's lack of documentation, leaving most parameters ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states 'Patch a template by id; only included fields change.' with a concrete example. The verb 'Patch' and resource 'template' are specific, and the scope is clear. It easily distinguishes from sibling tools like templates_create (create) and templates_delete (delete).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the description: it's for updating template fields. However, there is no explicit guidance on when to use this vs alternatives like templates_duplicate or templates_update, nor any prerequisites or restrictions. This leaves ambiguity for an AI agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

usage_getGet usage rollupA
Read-onlyIdempotent

Returns the team's prepaid free-run pool AND incurred Gemini-token spend over the requested window. The response distinguishes two separate concepts that must NOT be conflated: (1) freeRunsRemaining / freeRunsTotal — a fixed pool of prepaid autouser runs that covers Gemini token costs for ratings; once exhausted, every additional rating bills Gemini tokens to the team. (2) costUsd / per-run cost fields — the actual Gemini-token spend already incurred. A response showing $0 spent does NOT mean usage is free forever — it means either (a) no runs have happened in the window, or (b) the runs that did happen consumed the prepaid free pool. Once free runs are exhausted, each rating costs ~$0.091 (URL stimulus, SSE) to ~$0.137 (URL stimulus, SxS) based on stimulus type and eval type; one rating = one autouser × one comparison. When summarising this for a user, ALWAYS state both freeRunsRemaining and costUsd separately and explain that future runs will bill at the per-rating rate once the free pool is exhausted. Example: { range: '30d' }.

ParametersJSON Schema
NameRequiredDescriptionDefault
rangeNoTime window for the usage rollup. Defaults to '30d'.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds significant behavioral context beyond annotations: it clarifies that $0 spent does not mean free forever, explains the free pool exhaustion mechanism, and provides specific cost estimates per rating. This goes well beyond the readOnlyHint and idempotentHint annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is somewhat long but front-loads the core purpose and adds necessary caveats and cost details. Every sentence serves a purpose, though minor trimming could be possible. Well-structured overall.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simplicity of the tool (1 parameter, full schema coverage, output schema present), the description is fully complete. It explains the two separate concepts, cost implications, and how to present results. No gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter 'range' is fully documented in the input schema (100% coverage). The description does not add additional semantics beyond stating the default in an example. Baseline of 3 is appropriate since the schema already covers it adequately.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states what the tool does: 'Returns the team's prepaid free-run pool AND incurred Gemini-token spend over the requested window.' It distinguishes two separate concepts that must not be conflated, and provides an example. The purpose is specific and distinct from sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidelines: 'When summarising this for a user, ALWAYS state both `freeRunsRemaining` and `costUsd` separately and explain that future runs will bill at the per-rating rate once the free pool is exhausted.' It also explains how to interpret a $0 response. It does not explicitly state when not to use the tool, but the context makes it clear this is the only usage rollup tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 39 tool updatesv0.8.12
    • First observedautouser_run_get
    • First observedautouser_run_turns_list
    • First observedautouser_status_get
    • First observedautousers_calibration_freeze
    • First observedautousers_calibration_optimize
    • First observedautousers_calibration_start
    • First observedautousers_calibration_status_get
    • First observedautousers_create
    • First observedautousers_delete
    • First observedautousers_duplicate
    • First observedautousers_get
    • First observedautousers_list
    • First observedautousers_run
    • First observedautousers_run_stop
    • First observedautousers_update
    • First observedevaluations_agreement_get
    • First observedevaluations_ai_insights_get
    • First observedevaluations_create
    • First observedevaluations_delete
    • First observedevaluations_export_get
    • First observedevaluations_get
    • First observedevaluations_list
    • First observedevaluations_ratings_list
    • First observedevaluations_results_get
    • First observedevaluations_save_draft
    • First observedevaluations_share_create
    • First observedevaluations_shares_list
    • First observedevaluations_transfer
    • First observedevaluations_update
    • First observedteam_members_list
    • First observedteams_get
    • First observedteams_list
    • First observedtemplates_create
    • First observedtemplates_delete
    • First observedtemplates_duplicate
    • First observedtemplates_get
    • First observedtemplates_list
    • First observedtemplates_update
    • First observedusage_get

TDQS

A3.7/5.0

Scored across 39 tools

Disambiguation5/5

Every tool has a clear, distinct purpose with detailed descriptions. Tools are grouped by domain (autousers, evaluations, teams, templates, usage) and within each group, actions are clearly differentiated by verb suffixes (e.g., list, create, delete, get, update, etc.). There is no ambiguity between overlapping tool purposes.

Naming Consistency4/5

The naming convention is predominantly verb_noun with snake_case, following a predictable pattern. However, there is a minor inconsistency: some tools use 'autouser' (singular) as a prefix (e.g., autouser_run_get) while others use 'autousers' (plural) (e.g., autousers_create). This slight inconsistency prevents a perfect score.

Tool Count3/5

With 39 tools, the server covers multiple domains (autousers, evaluations, teams, templates, usage) comprehensively. While each domain has a reasonable number of tools, the overall count is high, bordering on heavy for a single server. The scope is broad, which may lead to a crowded tool surface, but each tool serves a necessary function.

Completeness5/5

The tool set provides complete CRUD/lifecycle coverage for autousers and evaluations, including creation, reading, updating, deletion, duplication, calibration, running, stopping, results retrieval, export, sharing, and transfer. Additionally, it covers team management, template management, and usage tracking, leaving no obvious gaps for the intended functionality.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    A
    maintenance
    ▎ Enables AI assistants to unleash a crowd of role-played virtual users — impatient shoppers, seniors, keyboard-only users, privacy hawks, chaos monkeys — on a website. Each persona browses in a real Chromium browser, files UX/QA findings in character, and the site receives a damage report with an S–F survival grade. Accusations can be cross-examined by up to three independent verification engines
    1
    MIT
  • F
    license
    Not graded
    quality
    C
    maintenance
    MCP server that exposes a design system and UX rulebook as tools, enabling AI to generate UI consistent with your design tokens, components, layout rules, accessibility guidelines, and templates. It also provides a review tool to score UI proposals against the design system.
    -