VerifyAX MCP
OfficialThe VerifyAX MCP server connects MCP clients to VerifyAX so you can manage agents, author scenarios, run evaluations, and inspect usage in natural language.
Register, list, and delete agents (A2A, API, DIRECTLINE, MCP), with optional connectivity checks.
Discover skill tags: list compatible tags, recommend tags via embeddings plus LLM, or search tags by natural-language query.
Generate scenarios (single or batch) with skill tags and context; list and delete scenarios.
Evaluate an agent against a scenario end-to-end, including cost preview, simulation run, and scores; list recent runs and get run details.
Preview credit cost for a planned run and summarize usage/spend over time or filtered by run, job, or scenario.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@VerifyAX MCPShow me the latest evaluation results for my agent"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
The official VerifyAX MCP Server is a bridge between your MCP-compatible AI client and the VerifyAX agent-evaluation platform. Once configured, it lets you register agents, generate scenarios, run simulations, and read evaluations in natural language — without writing API scripts.
With the VerifyAX MCP Server, you can:
Register and test agents (A2A or API) and confirm connectivity before evaluation.
Generate scenarios from skill tags and wait for completion in one tool call.
Run evaluations and read scores, transcripts, and credit usage without manual polling.
It complements (does not replace) the verifyax-api skill:
the skill is for developers writing code; the MCP server is for conversational workflows.
Contents
Related MCP server: Verix API MCP Server
Supported clients
The VerifyAX MCP Server works with MCP-compatible clients that support Streamable HTTP or stdio:
Client | Setup reference |
OpenAI ChatGPT | |
Claude (Claude.ai, Code, Desktop) | |
Cursor | |
Visual Studio Code (GitHub Copilot) | |
GitHub Copilot CLI | |
Google Gemini CLI | |
Amazon Quick Suite |
Any client that can connect via mcp-remote can also
use the hosted endpoint at https://mcp.verifyax.com/mcp.
Claude.ai custom connectors: use No sign-in and an Authorization header. This server
authenticates with a VerifyAX API key, not OAuth, so in Settings → Connectors → Add custom
connector:
URL:
https://mcp.verifyax.com/mcpAuthentication: No sign-in — described in the dialog as "for servers that use an API key instead of OAuth". Leave Sign in now unselected; this server publishes no OAuth metadata, so an OAuth attempt fails.
Under Request headers, add
Authorizationwith the valueBearer sk-ver-api-....
All 17 tools then appear, and Claude.ai lets you set approval per tool (it groups them as 12
read-only and 5 write/delete — worth setting the write/delete group to require approval, since
generate_scenario and evaluate_agent spend credits).
Requires server 0.3.5 or later: earlier builds matched the Bearer scheme case-sensitively
and rejected clients that normalise it, which surfaced only as "Couldn't connect to the server."
The X-VerifyAX-API-Key header works with the server but Claude.ai's header field rejects that
name, so use Authorization.
For step-by-step Claude setup and how this compares to the VerifyAX skill and SDK, seedocs/using-verifyax-with-claude.md.
Supported tools
Seventeen tools mapped to user intents. Blocking tools poll internally and return only when work completes (typically 30s–5min).
Area | Tools | Blocking |
Agents |
| — |
Scenarios |
| yes ( |
Evaluation |
| yes ( |
Usage |
| — |
For tool descriptions (what Claude reads to pick a tool) and rationale, seedocs/tool-descriptions.md. Package-level reference:
packages/mcp-server/README.md.
Before you start
Requirements depend on how you connect.
Remote HTTP (hosted at mcp.verifyax.com)
A VerifyAX API key (Settings → API Keys in the console)
An MCP client with Streamable HTTP support, or Node.js 18+ to run the
mcp-remoteproxy
Local stdio (verifyax-mcp-server via npx)
Node.js ≥ 20
A VerifyAX API key in your client config (
VERIFYAX_API_KEY)
The npm package exposes two binaries (verifyax-mcp-server and verifyax-mcp-server-http); pass
the stdio binary name explicitly when using npx.
Install the MCP server
Remote HTTP (recommended)
Hosted endpoint:
https://mcp.verifyax.com/mcpSend your VerifyAX API key on the initialize request. The hosted server does not store keys — each client session brings its own.
Cursor (native URL — v0.48+):
{
"mcpServers": {
"verifyax": {
"url": "https://mcp.verifyax.com/mcp",
"headers": {
"Authorization": "Bearer sk-ver-api-..."
}
}
}
}Do not add a transport field — Cursor detects Streamable HTTP from the URL. You can also use
"X-VerifyAX-API-Key": "sk-ver-api-..." instead of Authorization.
Claude Desktop / clients without native HTTP (mcp-remote):
Add to claude_desktop_config.json (Settings → Developer → Edit Config):
{
"mcpServers": {
"verifyax": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.verifyax.com/mcp",
"--transport",
"http-only",
"--header",
"Authorization:${VERIFYAX_AUTH}"
],
"env": {
"VERIFYAX_AUTH": "Bearer sk-ver-api-..."
}
}
}
}Use --transport http-only because this server speaks Streamable HTTP only (no legacy SSE). You
can also pass X-VerifyAX-API-Key: sk-ver-api-... via --header instead of Authorization.
Restart your MCP client after changing config.
Local stdio
Claude Code:
claude mcp add verifyax --env VERIFYAX_API_KEY=sk-ver-api-... -- npx -y -p @verifyax/mcp-server verifyax-mcp-serverThe version here is deliberately unpinned, so this command always fetches the current server.
Only the verifyax-mcp plugin pins an exact
version (@verifyax/mcp-server@<plugin version>), because a marketplace plugin promises that users
get a new server build only when the plugin version is bumped. Setting up by hand carries no such
promise, so tracking the latest release is the right default. Pin it yourself if you need
reproducibility: -p @verifyax/mcp-server@0.3.5.
-p … verifyax-mcp-server names the binary explicitly because the package ships two
(verifyax-mcp-server, verifyax-mcp-server-http) and neither matches the package name, so a
bare npx -y @verifyax/mcp-server fails with "could not determine executable to run".
Claude Desktop:
{
"mcpServers": {
"verifyax": {
"command": "npx",
"args": ["-y", "-p", "@verifyax/mcp-server", "verifyax-mcp-server"],
"env": { "VERIFYAX_API_KEY": "sk-ver-api-..." }
}
}
}Restart the client after changing config, then describe what you want in natural language.
How it works
Architecture and communication
Your MCP client connects to
https://mcp.verifyax.com/mcp(remote HTTP) or spawnsnpx -p @verifyax/mcp-server verifyax-mcp-server(local stdio).The client sends your VerifyAX API key on initialize (
Authorization: Bearer …orX-VerifyAX-API-Keyfor HTTP;VERIFYAX_API_KEYenv for stdio).The server calls the VerifyAX API on your behalf. Blocking tools (
generate_scenario,evaluate_agent) poll job status internally and return only when work finishes.
Permission and billing
Actions are scoped to the workspace tied to your API key.
Usage and credits are billed to your VerifyAX workspace, not to the MCP server operator.
The hosted server sends no telemetry and does not persist API keys between sessions.
Self-hosting
Run the same HTTP server on your own infrastructure (e.g. Google Cloud Run). See
deploy/gcp/README.md.
Example workflows
Once connected, describe tasks in natural language — the client picks the tool.
Agent setup
Register: "Register my A2A agent at https://my-agent.example.com and confirm it's reachable."
List: "What agents are registered in my workspace?"
Scenario authoring
Discover tags: "List the skill tags I can use for an interview scenario."
Generate: "Generate an info_exchange scenario tagged empathy and coordination."
Evaluation
Run eval: "Evaluate agent X against scenario Y and summarize the scores."
Review history: "Show details for my most recent simulation run."
Usage and cost
Preview: "How many credits will it cost to run this scenario against my agent?"
Summary: "What did my last 5 simulation runs cost?"
Blocking tools can take 30s–5min. Do not call them repeatedly — wait for the result.
Tips and tricks
Add defaults to AGENTS.md
Reduce discovery calls and bad tag combinations by adding this to an AGENTS.md file in your
project root (see the AGENTS.md convention for the format):
## VerifyAX MCP
When connected to verifyax:
- **MUST** call `recommend_scenario_tags`, `search_scenario_tags`, or `list_compatible_tags` before `generate_scenario`
- **MUST NOT** combine QnA tags with other tags (QnA must be the sole tag)
- **MUST** use `preview_run_cost` when the user asks about credits before `evaluate_agent`
- Blocking tools (`generate_scenario`, `evaluate_agent`) take 30s–5min — do not retry manuallyUse the skill for code workflows
For scripts, CI, or custom multi-step logic, use the
verifyax-api skill or
@verifyax/sdk instead of MCP tools.
Data and security
All traffic to the hosted endpoint uses HTTPS (TLS).
API key authentication; every action respects your VerifyAX workspace permissions.
The hosted server does not store API keys — each session supplies its own.
The MCP server sends no telemetry; it talks only to the VerifyAX API.
Destructive tools (
delete_agent,delete_scenario) permanently remove resources — confirm with the user before calling them.
API keys are managed in the VerifyAX console (Settings → API Keys). Revoke a key there to cut off access immediately.
Troubleshooting
"VERIFYAX_API_KEY is not set" — the key isn't reaching the server. Check the
envblock (stdio) orAuthorization/X-VerifyAX-API-Keyheader (HTTP).Authentication failed — the key is invalid, revoked, or from the wrong environment. Mint a fresh one in the console.
Tool calls don't appear — MCP clients load tools at startup; restart the client (or start a new session) after adding the server.
HTTP connection fails in Cursor — try
mcp-remotewith--transport http-only, or verify the URL ends with/mcp.Want logs? Set
VERIFYAX_MCP_LOG_LEVEL=debug. Logs are structured JSON on stderr; stdout is reserved for the MCP protocol.
Support and feedback
This project is maintained by the VerifyAX team at Conscium.
Bugs and feature requests: open an issue
Development reference:
CONTRIBUTING.md(maintainers and forks)External pull requests aren't accepted — issues are the best way to reach us
Disclaimer
MCP clients can register agents, generate scenarios, run evaluations, and delete resources using your VerifyAX API key and workspace credits. Use least privilege (scoped keys, minimal workspace access), review high-impact actions before confirming, and revoke keys you no longer need.
Model Context Protocol connects AI agents to external tools, which creates powerful workflows but also structural risks. Large language models can be vulnerable to prompt injection and related attacks. Only use trusted MCP clients and servers, and review which tools each agent can access.
For developers
This repository is a pnpm monorepo with two packages:
Package | Description |
Typed TypeScript client for the VerifyAX REST API | |
MCP server (17 tools) built on the SDK |
Requires Node ≥ 20 and pnpm 10.
pnpm install # install workspace dependencies
pnpm build # build all packages (topological order)
pnpm test # run unit tests
pnpm lint # lint
pnpm format # format with prettierNetwork-dependent suites: pnpm test:integration (live API, needs VERIFYAX_TEST_KEY) and
pnpm test:conformance (spawns the built MCP server over stdio).
MCP Inspector:
docs/debugging-mcp-inspector.mdExecution plan:
docs/PLAN.mdArchitecture decisions:
CLAUDE.mdCloud Run deploy:
deploy/gcp/README.md
License
Apache-2.0. See LICENSE.
Available Tools
14 toolsdelete_agentDelete agentADestructiveIdempotentInspect
Permanently deletes an agent from your VerifyAX workspace by its uuid. This cannot be undone. Returns confirmation of the deletion.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_uuid | Yes | The uuid of the agent to permanently delete. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true and idempotentHint=true, so safety is covered structurally. The description adds genuine value beyond that by warning the deletion is permanent and cannot be undone, and by noting a confirmation is returned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short, front-loaded sentences with no filler; each adds a distinct fact (what, irreversibility, return confirmation).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter destructive tool with annotations covering safety and no output schema, the description supplies the irreversibility warning and return confirmation, leaving little an agent needs missing. Minor gap: no permissions or preconditions context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Only one parameter, and schema description coverage is 100% with the uuid semantics fully documented in the schema. The description merely restates that the agent is identified by uuid, adding no syntax or format detail, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (deletes) and resource (agent) with the identifying key (uuid) and workspace scope. It is distinguishable from delete_scenario by the named resource, though it never explicitly contrasts with that sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this versus alternatives such as list_agents or register_agent, and no prerequisites stated (e.g. permissions, required prior state). The agent must infer the calling context entirely.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_scenarioDelete scenarioADestructiveIdempotentInspect
Permanently deletes a scenario from your VerifyAX workspace by its uuid. This cannot be undone and fails if simulation runs still reference the scenario. Returns confirmation of the deletion.
| Name | Required | Description | Default |
|---|---|---|---|
| scenario_uuid | Yes | The uuid of the scenario to permanently delete. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true and idempotentHint=true, yet the description still adds meaningful context: the deletion is permanent and cannot be undone, and it is blocked while simulation runs reference the scenario. The only omission is confirmation of return shape, which is mentioned but not detailed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with the action and resource, then the irreversibility/constraint, then the response. No filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter destructive tool, the description covers permanence, the referential failure mode, and the return outcome. Nothing an agent needs before invoking it is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
One parameter with 100% schema description coverage; the schema already documents scenario_uuid fully. The description only restates that the uuid identifies the scenario, adding no format or sourcing detail beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (permanently deletes), resource (scenario), and scope (by uuid, in the VerifyAX workspace). It is clearly distinguishable from sibling delete_agent and from read/list tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear precondition for use and failure: it fails if simulation runs still reference the scenario. It does not name an alternative tool or spell out when-not-to-use beyond that precondition, but the context is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
evaluate_agentEvaluate agentAInspect
Runs an agent against a scenario and evaluates the result end to end. Typically takes 30s–30min; task-capable MCP clients receive a pollable task handle immediately, while others block until the evaluation completes. Give it an agent uuid and a scenario uuid; it previews cost, runs the simulation, waits for it, and returns the evaluation scores. Optional timeout_minutes (1–240) overrides the scenario default for this run.
| Name | Required | Description | Default |
|---|---|---|---|
| num_runs | No | Parallel repetitions, 1-10 (default 1). | |
| agent_uuid | Yes | The agent to evaluate. | |
| scenario_uuid | Yes | The scenario to run the agent against. | |
| timeout_minutes | No | Wall-clock budget in minutes for this run (overrides scenario default). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=false and destructiveHint=false, so the description carries the real behavioral load and delivers: 30s–30min runtime, pollable task handle for task-capable clients vs blocking behavior for others, cost preview before the run, and the wait-then-return-scores flow. This is exactly the operational context an agent needs before committing to a long call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the action and duration, followed by client-behavior detail and input requirements. It is tight and well-ordered; the 'give it an agent uuid and a scenario uuid' clause mildly duplicates the schema but reads as a necessary lead-in to the workflow.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a long-running tool with no output schema, the description still conveys the execution model (async task handle vs blocking), expected latency, cost preview, and the fact that evaluation scores are returned. Nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all four parameters are already documented, including num_runs and the 1–240 range on timeout_minutes. The description's mention of timeout_minutes overriding the scenario default restates what the schema already says, adding no new syntax or format detail. Baseline 3 applies when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource chain ('runs an agent against a scenario and evaluates the result end to end') and enumerates what it does (preview cost, run simulation, wait, return scores). It is clearly distinct from get_run_details or preview_run_cost by implication, but it never names a sibling or explicitly excludes one, so it stops short of full sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: the agent is told what inputs to supply ('give it an agent uuid and a scenario uuid') but not when to choose this tool over preview_run_cost or get_run_details. No exclusions or prerequisites are given, leaving routing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
generate_scenarioGenerate scenarioAInspect
Generates a new test scenario of a given type (info_exchange or interview) with optional skill tags and context. Invalid tag names or incompatible tags fail immediately with 400/422 before a job is queued; use recommend_scenario_tags, search_scenario_tags, or list_compatible_tags to pick valid tag names. Typically takes 30s–2min; task-capable MCP clients receive a pollable task handle immediately, while others block until generation finishes. Set num_scenarios greater than 1 for batch mode (requires tag_pool). Returns the new scenario’s uuid, or batch uuids when batching, or a structured error if generation fails for other reasons after the job starts.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Workspace-unique scenario name. | |
| tags | No | Skill tag names (use recommend_scenario_tags, search_scenario_tags, or list_compatible_tags). | |
| tag_pool | No | Required when num_scenarios > 1; universe of tag names to sample from. | |
| total_tags | No | Batch only; tags drawn per scenario from tag_pool. | |
| include_tags | No | Batch only; tags required in every scenario (subset of tag_pool). | |
| num_scenarios | No | Batch size. 1 = single scenario (default). Greater than 1 requires tag_pool. | |
| scenario_type | Yes | ||
| context_prompt | No | ||
| max_tags_per_npc | No | Batch only; caps tags per NPC. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare only that this is not read-only and not destructive. The description adds substantial context beyond that: validation failures return 400/422 synchronously before any job is queued, typical latency is 30s–2min, task-capable clients get a pollable handle while others block, and post-start failures return a structured error.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the core purpose and validation behavior, then layers timing, async semantics, batch mode, and return values in a logical order. Each sentence carries information, though the density of caveats makes it heavier than ideal.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, and the description compensates by describing the return value (scenario uuid, batch uuids, or structured error). Combined with validation behavior, latency, and async handling, an agent has everything needed to call this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 78%, so most parameters are documented structurally, but the description adds cross-parameter semantics the schema does not: num_scenarios > 1 requires tag_pool, and that mode is called "batch mode". It does not explain total_tags, include_tags, or max_tags_per_npc beyond what the schema already states, so it stops short of full compensation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource ("Generates a new test scenario") and immediately enumerates the scenario_type values (info_exchange or interview). It clearly differentiates itself from the sibling tag-discovery tools by naming them as inputs to this operation rather than alternatives to it.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: use recommend_scenario_tags, search_scenario_tags, or list_compatible_tags to pick valid tag names. It states the condition for batch mode (num_scenarios > 1 requires tag_pool) and explains the client-dependent blocking vs task-handle behavior, so an agent knows exactly what to expect.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_run_detailsGet run detailsARead-onlyInspect
Fetches the full details of a single simulation run by its uuid, including its status and the evaluation results when they are available. Use after a run to inspect scores and outcome.
| Name | Required | Description | Default |
|---|---|---|---|
| simulation_uuid | Yes | The run’s uuid. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe read, so the bar is lower. The description adds real value by disclosing that evaluation results are conditional ('when they are available'), warning the agent that scores may be absent for an in-progress run. It says nothing about error behavior for bad/unknown uuids.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the purpose front-loaded and the usage note second. The second sentence mildly restates the first ('inspect scores and outcome' vs 'status and the evaluation results'), a small redundancy but not bloat.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description must carry the return-value burden, and it does name the key fields (status, evaluation results) plus their conditional availability. For a one-param read tool this is essentially sufficient; only uuid error handling is unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and there is a single parameter whose description ('The run's uuid.') matches what the tool description conveys. The description adds no format, constraint, or lookup guidance beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Fetches') and resource ('full details of a single simulation run'), with the identifier ('by its uuid') and expected payload (status, evaluation results). It implicitly separates itself from the list_recent_runs sibling by scoping to a single run, but never names an alternative explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Use after a run to inspect scores and outcome' gives clear timing and intent for when to reach for this tool. It stops short of naming the alternative (list_recent_runs) or stating when not to use it, so it is context without full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_usage_summaryGet usage summaryBRead-onlyInspect
Summarizes VerifyAX usage events over an optional time range or for a specific simulation, scenario, or job. Paginates across all matching events (up to a cap) and returns the total event count, a breakdown by product area, and total platform spend in USD when the API reports it.
| Name | Required | Description | Default |
|---|---|---|---|
| failed | No | ||
| job_uuid | No | Filter by async job UUID. | |
| max_events | No | Cap on events to summarize (default 10000). | |
| product_area | No | ||
| scenario_uuid | No | ||
| event_start_to | No | ISO 8601 end of the window. | |
| simulation_uuid | No | ||
| event_start_from | No | ISO 8601 start of the window. | |
| evaluation_job_uuid | No | Filter by evaluation job UUID. | |
| simulation_job_uuid | No | Filter by simulation job UUID. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe read. The description adds a genuinely useful behavioral detail that annotations do not cover: pagination across all matching events up to a cap, and that spend is returned only 'when the API reports it' – an important conditional the agent should know about. It still omits how the cap interacts with max_events and whether partial results are signaled.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three clauses in a single tight sentence, front-loading the summary action, then scoping, then return contents. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter read-only aggregator with no output schema, the description covers purpose, scope, pagination, and the shape of the return value well. It stops short of explaining the four undescribed parameters or the difference between the various *_uuid filters, which is the main remaining gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 60%, so half the parameters (failed, product_area, scenario_uuid, simulation_uuid) have no inline descriptions. The prose gestures at the filter dimensions but does not explain semantics for the undocumented parameters, leaving the description to partially compensate rather than fully cover the gap. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear verb+resource (summarizes VerifyAX usage events) and even names the output fields (event count, breakdown by product area, platform spend). It is not explicitly differentiated from the nearest sibling preview_run_cost, but the purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions the filter dimensions obliquely ('over an optional time range or for a specific simulation, scenario, or job') but never states when to use this tool versus preview_run_cost or other cost-related siblings, nor any prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_agentsList agentsBRead-onlyInspect
Lists the AI agents registered in your VerifyAX workspace, optionally filtered by connector type (A2A, API, DIRECTLINE, EXTENSION, or MCP). Returns each agent’s uuid, name, type, and URL.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| offset | No | ||
| agent_type | No | Filter to a single agent type. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe read, so the description's burden is lower. It adds useful context by enumerating the filter values and the returned fields (uuid, name, type, URL), but says nothing about pagination behavior or result ordering despite exposing limit/offset.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences: the core action and optional filter come first, the return shape second. No filler, no repetition beyond the necessary enum listing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Since there is no output schema, the description correctly covers the return fields, which is exactly what an agent needs. The remaining gap is pagination semantics for limit/offset, the only substantive omission for a list tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 33% – agent_type is documented in the schema, while limit and offset have no description anywhere. The description merely restates the enum values for agent_type, an optional filter, and never explains the pagination parameters, so it fails to compensate for the documentation gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and resource ("Lists the AI agents registered in your VerifyAX workspace") plus the filter dimension, so it is unambiguous what the tool returns. It does not explicitly contrast itself with siblings like register_agent or delete_agent, but the resource is distinct enough that confusion is unlikely.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description notes the filter is optional and names the allowed connector types, which implies the use case (browsing/filtering registered agents). However, it never states when to prefer this over other discovery tools or what happens when no filter is supplied, leaving usage to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_compatible_tagsList compatible skill tagsARead-onlyInspect
Lists the full skill-tag catalogue filtered to tags compatible with a given scenario type (info_exchange or interview). Use recommend_scenario_tags or search_scenario_tags for a ranked shortlist when the user gave natural language or the list is too large. Returns each tag’s name, category, and description, and flags QnA tags that must be the only tag.
| Name | Required | Description | Default |
|---|---|---|---|
| scenario_type | Yes | The kind of scenario the tags will be used for. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe read. The description goes further by disclosing the return contents (name, category, description) and a domain-specific constraint — QnA tags flagged as needing to be the only tag — which is behavioral context not present in the annotations or schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences: purpose and filter up front, then the alternative-tool routing, then the return summary. No filler, and the most decision-relevant information (what it returns, which sibling to use) leads.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, yet the description covers the return shape and the special QnA flag, and an annotation covers the safety profile. For a one-parameter read-only listing tool, nothing needed to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single enum parameter is fully documented, so the baseline is 3. The description restates the two enum values (info_exchange, interview) but adds no syntax, format, or defaulting detail beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Lists) and resource (full skill-tag catalogue) with explicit scope: filtered to tags compatible with a given scenario type. It also names the sibling tools it is not (recommend_scenario_tags, search_scenario_tags), so an agent can distinguish it without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit alternative and the selecting condition: use recommend_scenario_tags or search_scenario_tags for a ranked shortlist when the user gave natural language or the list is too large. This is a clear when-to-use-this-vs-others rule.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_recent_runsList recent runsARead-onlyInspect
Lists recent simulation runs in your VerifyAX workspace, optionally filtered by status, agent, scenario, date range, search text, or run group. Returns each run’s uuid, status, agent, scenario, and evaluation handle.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| offset | No | ||
| search | No | Free-text search across run metadata. | |
| status | No | Filter by run status. Known values: CREATED, IN_PROGRESS, COMPLETED, FAILED, CANCELLED (open enum — the API may add statuses, which are forwarded rather than rejected). | |
| date_to | No | ISO 8601 end of the created-at window. | |
| date_from | No | ISO 8601 start of the created-at window. | |
| agent_uuid | No | ||
| scenario_uuid | No | ||
| run_group_uuid | No | Filter to runs in a linked run group. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already declares the safety profile, so the description's main added value is naming the returned fields (uuid, status, agent, scenario, evaluation handle). It does not mention pagination behavior despite limit/offset, and adds no rate-limit or ordering context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly written sentences with no filler; the purpose is front-loaded and the filter list and return summary follow logically.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description does the right thing by enumerating the returned fields. The only material gap is the absence of any pagination (limit/offset) or result-ordering note for what is clearly a paged listing tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 56%, so roughly half the parameters carry no inline documentation. The description compensates by naming the filterable dimensions (status, agent, scenario, date range, search text, run group), matching the schema fields but adding no syntax or format detail beyond them.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Lists) and resource (recent simulation runs) scoped to a workspace, and enumerates the available filter dimensions. An agent can distinguish this bulk-list tool from the singular get_run_details sibling without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The word 'optionally filtered' implies this is a browse/query tool, but it never states when to reach for it versus get_run_details or how the filters combine. Usage is implied rather than directed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_scenariosList scenariosARead-onlyInspect
Lists the test scenarios in your VerifyAX workspace, optionally filtered by type (info_exchange or interview) and status. Returns each scenario’s uuid, name, type, and status.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| offset | No | ||
| status | No | Filter by scenario status. Known values: INIT, PROCESSING, SUCCESS, FAILED, CANCELLED (open enum — the API may add statuses, which are forwarded rather than rejected). | |
| scenario_type | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The readOnlyHint annotation already covers the safety profile, so the description earns credit for disclosing the return fields (uuid, name, type, status). However it says nothing about pagination behavior despite limit/offset existing, leaving result-set handling opaque.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with zero waste. The scope and filters are front-loaded and the return shape follows immediately, so every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only list tool with no output schema, the description usefully names the returned fields. It is nearly complete, with the only gap being the unexplained limit/offset paging semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 25%: status is documented, scenario_type has an enum, but limit and offset are bare in both schema and description. The description does reinforce the type values (info_exchange/interview) and the status filter, but does not compensate for the undocumented pagination parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Lists') and resource ('test scenarios') scoped to the VerifyAX workspace, and enumerates the supported filters. It is clear and self-contained, though it doesn't explicitly contrast itself with sibling list/search tools like search_scenario_tags.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'optionally filtered by type ... and status' implies the usage context, but there is no explicit when-to-use vs when-not guidance and no alternative tool named. The agent must infer that this is the unfiltered enumeration tool rather than a search sibling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
preview_run_costPreview run costARead-onlyInspect
Estimates the credit cost of running an agent against a scenario before triggering it. Returns the estimated credits, your current balance, and any pending committed spend. Optional timeout_minutes (1–240) affects the run-cost estimate.
| Name | Required | Description | Default |
|---|---|---|---|
| num_runs | No | Parallel repetitions (default 1). | |
| agent_uuid | No | The agent that would run (optional). | |
| scenario_uuid | Yes | The scenario the run would use. | |
| timeout_minutes | No | Wall-clock budget in minutes for the estimate. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The readOnlyHint annotation already covers the safety profile, so the description's added value is the disclosure of what is returned: estimated credits, current balance, and pending committed spend. It also notes that timeout_minutes feeds into the estimate. It stops short of stating whether estimates are guaranteed or how stale the balance figure is.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences: purpose and timing first, then return payload, then the one parameter worth calling out. No filler, no repetition of the name or title, and the most decision-relevant clause is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, and the description compensates by enumerating the three return values. Combined with a complete parameter schema, full annotation coverage of the read-only nature, and the stated pre-run timing, an agent has everything needed to call this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3 per the rubric. The description restates timeout_minutes and its 1-240 range and adds the causal note that it 'affects the run-cost estimate', which is marginally useful, but it adds nothing about num_runs or agent_uuid beyond what the schema already documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Estimates the credit cost of running an agent against a scenario') plus the key temporal qualifier 'before triggering it'. No sibling tool (evaluate_agent, get_run_details, get_usage_summary) performs pre-flight cost estimation, so the agent can distinguish it immediately.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'before triggering it' clearly establishes the pre-flight use context, which is the main decision point for an agent choosing this tool. However, it names no alternative and gives no explicit when-not condition (e.g., that this is read-only and safe to call speculatively), so it falls short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
recommend_scenario_tagsRecommend scenario skill tagsARead-onlyInspect
Suggests skill tags for a scenario type using the same Workbench pipeline as the Scenario Generator (embeddings plus LLM, up to about 20 tags). Use when the user describes a scenario or agent and wants a ranked shortlist before generate_scenario. Prefer this over list_compatible_tags when the catalogue is large or the user gave natural-language context. Read-only (no generation credits). Provide context_prompt and/or agent_uuid (at least one). On recommender timeout or unavailability, try search_scenario_tags or list_compatible_tags.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_uuid | No | Optional workspace agent uuid; A2A agents may trigger a live agent-card fetch. | |
| scenario_type | Yes | The kind of scenario the tags will be used for. | |
| context_prompt | No | Optional scenario context (same meaning as generate_scenario context_prompt). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint, and the description adds real context: 'Read-only (no generation credits)', the embedding+LLM pipeline, a ~20 tag cap, and timeout/unavailability behavior. It stops short of describing latency or the exact ranking semantics, so it is strong but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose then routing then prerequisites then fallbacks; every sentence carries information. It is dense and slightly over-stuffed for a 3-parameter tool, but nothing is redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description does convey the return shape (ranked shortlist of up to ~20 tags) plus failure handling, so an agent can call and recover correctly. Minor gaps remain around what happens when neither optional param yields a match.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3, but the description adds a constraint the schema does not encode: 'Provide context_prompt and/or agent_uuid (at least one)', which clarifies that scenario_type alone is technically valid but semantically insufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Suggests skill tags for a scenario type') and names the mechanism (Workbench pipeline, embeddings plus LLM, up to ~20 tags). It is immediately distinguishable from siblings like list_compatible_tags and search_scenario_tags.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use ('user describes a scenario or agent and wants a ranked shortlist before generate_scenario'), explicit preference over list_compatible_tags for large catalogues or natural-language context, and explicit fallbacks on timeout ('try search_scenario_tags or list_compatible_tags'). This is a complete routing guide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
register_agentRegister agentAInspect
Registers an AI agent in VerifyAX given its name, connector type (A2A, API, DIRECTLINE for Copilot Studio, or MCP), URL, and optional auth or connector settings. For A2A agents it first verifies the agent card is reachable; DIRECTLINE and MCP agents are probed before creation. Returns the new agent’s uuid and whether connectivity was checked. Note: any token, Direct Line secret, or password passed here transits the conversation — prefer supplying credentials out of band where possible.
| Name | Required | Description | Default |
|---|---|---|---|
| mcp | No | ||
| name | Yes | Workspace-unique agent name. | |
| token | No | Token for bearer/cs auth. Sensitive — see the note in the tool description. | |
| timeout | No | Request timeout in milliseconds. | |
| agent_url | No | Endpoint URL. Required for A2A, API, and MCP (catalogue adapter A2A URL). Optional for DIRECTLINE — derived from region when omitted. | |
| agent_type | No | Defaults to A2A. | |
| directline | No | ||
| auth_method | No | ||
| description | No | ||
| agent_card_url | No | ||
| basic_password | No | Sensitive — see the note in the tool description. | |
| basic_username | No | ||
| agent_card_path | No | ||
| default_output_modes | No | ||
| include_full_context | No | ||
| include_message_history | No | ||
| max_requests_per_minute | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=false and destructiveHint=false, so the description carries the interesting behavior and does so well: it discloses the pre-creation reachability check for A2A and the probing for DIRECTLINE/MCP, states the return payload (new uuid plus whether connectivity was checked), and warns that credentials transiting the conversation should be supplied out of band. That is meaningful side-effect and security context beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences, front-loaded with the core action and connector enumeration before the behavioral and security notes. Dense but each sentence contributes; only slightly longer than strictly necessary.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 17-parameter mutation tool with no output schema, the description supplies the return values, connector-specific behavior, and a credential-handling caveat, which covers the agent's main decision points. Remaining gaps are the undocumented optional tuning parameters, which the 35% schema coverage does not fill in either.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 35% across 17 parameters, so the description is expected to compensate. It identifies the primary inputs (name, connector type with its enum values, URL, optional auth/connector settings), which covers the essential fields, but leaves many optional parameters (timeout, include_full_context, max_requests_per_minute, default_output_modes, agent_card_path, etc.) unexplained in either place.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb (registers) and resource (AI agent in VerifyAX), then enumerates the four connector types (A2A, API, DIRECTLINE, MCP) that shape the call. This cleanly distinguishes it from the read/delete siblings such as list_agents and delete_agent without the agent needing to open any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explains what happens per connector type (A2A card verification, DIRECTLINE/MCP probing) but never states when to prefer this tool over alternatives or any prerequisites/exclusions. Usage is implied by the connector-type branching rather than made explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_scenario_tagsSearch scenario skill tagsARead-onlyInspect
Finds skill tags for a scenario type by embedding similarity to a natural-language query (no LLM). Cheaper than recommend_scenario_tags — good for “find tags about X”. Read-only. A per-user tag-search rate limit applies in addition to the workspace public API limit; on rate limit, honor Retry-After, wait and retry, or use list_compatible_tags for the full filtered catalogue.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max tag names to return (default 50). | |
| query | Yes | Natural-language search text (at least 2 characters). | |
| scenario_type | Yes | The kind of scenario the tags will be used for. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and the description confirms read-only, so no contradiction. It adds genuinely useful behavior beyond annotations: a per-user tag-search rate limit on top of the workspace API limit, plus concrete handling ('honor Retry-After, wait and retry'). It stops short of describing the return shape, but the rate-limit disclosure is above the bar. Note: no output schema exists, so return details would have been a bonus.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences, front-loaded with what it does and how it compares to the sibling, then the operational caveat. Every sentence earns its place, though the rate-limit clause is packed densely.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only search tool with a fully described schema, the definition covers purpose, alternatives, and the rate-limit failure path, which is what an agent needs to invoke it correctly. The only gap is return-shape detail, minor because results are implicitly tag names and no output schema is declared.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters (scenario_type enum, query natural-language text, limit) are already documented in the schema. The description adds no syntax or format detail beyond what the schema provides, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource (finds skill tags), the mechanism (embedding similarity to a natural-language query, no LLM), and the scoping dimension (scenario type). It also distinguishes itself from the sibling recommend_scenario_tags by cost, so an agent can tell the two apart without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly names when to use it ('find tags about X', cheaper than recommend_scenario_tags) and states the fallback for the failure case ('use list_compatible_tags for the full filtered catalogue'). Alternatives and conditions are spelled out rather than left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
14 tool updates
v0.3.7- Changed
delete_agent1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
delete_scenario1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
evaluate_agent1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
generate_scenario3 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_tags_per_npc / maximumAdded value: +9007199254740991 - changed
Input schema / properties / tags / descriptionPrevious value: -"Skill tag names (use list_compatible_tags to pick valid ones)."New value: +"Skill tag names (use recommend_scenario_tags, search_scenario_tags, or list_compatible_tags)."
- Changed
get_run_details1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
get_usage_summary2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_events / maximumAdded value: +9007199254740991
- Changed
list_agents2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / offset / maximumAdded value: +9007199254740991
- Changed
list_compatible_tags1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
list_recent_runs2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / offset / maximumAdded value: +9007199254740991
- Changed
list_scenarios2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / offset / maximumAdded value: +9007199254740991
- Changed
preview_run_cost2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / num_runs / maximumAdded value: +9007199254740991
- Added
recommend_scenario_tags - Changed
register_agent5 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / directline / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_requests_per_minute / maximumAdded value: +9007199254740991 - removed
Input schema / properties / mcp / additionalPropertiesRemoved value: -false - added
Input schema / properties / timeout / maximumAdded value: +9007199254740991
- Added
search_scenario_tags
3 tool updates
v0.3.5- Added
delete_agent - Added
get_usage_summary - Added
list_scenarios
7 tool updates
v1.0.1- Removed
delete_agent - Added
delete_scenario - Added
generate_scenario - Removed
get_usage_summary - Added
list_agents - Removed
list_scenarios - Added
register_agent
8 tool updates
v1.0.0- First observed
delete_agent - First observed
evaluate_agent - First observed
get_run_details - First observed
get_usage_summary - First observed
list_compatible_tags - First observed
list_recent_runs - First observed
list_scenarios - First observed
preview_run_cost
TDQS
Scored across 14 tools
The three tag-discovery tools (list_compatible_tags, recommend_scenario_tags, search_scenario_tags) share an overlapping goal but their descriptions clearly differentiate full-catalogue filtering, LLM-ranked recommendation, and embedding-based search. Most other tools target distinct resource/action pairs (scenario, agent, run, cost/usage).
All tool names use a consistent snake_case verb_noun pattern: delete_scenario, list_agents, register_agent, generate_scenario, evaluate_agent, get_run_details, preview_run_cost, etc. Deviations are minor and still follow the same convention.
14 tools is well within the ideal 3–15 range for a platform covering agents, scenarios, tags, runs, evaluation, and usage. Each tool appears to earn its place by covering a distinct lifecycle operation or query type.
Core create/list/delete operations exist for agents and scenarios, plus run evaluation, cost preview, and usage summary. However, update operations for agents and scenarios are missing, and there is no get_scenario or get_agent detail tool—only list endpoints—creating notable gaps for managing existing resources.
Maintenance
Related MCP Connectors
Test the voice agents you run: scored transcripts, pass/fail verdicts, latency and WER metrics.
- OkareoOAuthcom.okareo
Simulation, evaluation and monitoring for voice agents.
Context and control for any AI. Agent identity, scoped access, and proof across tools.
Manage Speko voice-AI agents, sessions, calls, phone numbers, knowledge bases, evals, and docs.
Related MCP Servers
AlicenseAqualityBmaintenanceEnables AI assistants to interact with Coval's evaluation platform for launching and monitoring evaluation runs, managing agents and test sets, and retrieving evaluation metrics.1814 npm3MIT- AlicenseNot gradedqualityCmaintenanceEnables searching Verix records, retrieving record details, and running analysis workflows through natural language via the Verix API.MIT
- AlicenseNot gradedqualityBmaintenanceEnables LLM evaluation and observability by uploading documents, building test sets, running RAG pipelines, and automatically scoring answers for groundedness, hallucination risk, retrieval quality, latency, and cost, with tools exposed to MCP-compatible clients.1MIT
- FlicenseNot gradedqualityDmaintenanceEnables SQL query execution and knowledge base search via RAG agent through two tools.-