Server Quality Checklist

Tool Scores

• list_runsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It does not disclose whether the operation is read-only, destructive, or requires specific permissions. The phrase 'connected to the daemon' hints at a dependency but lacks details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, appropriate for a tool with no parameters. It is front-loaded and directly states the action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description lacks completeness. It does not explain what 'active' means, the output format, or how results are returned. For a simple list tool, more context would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, and schema coverage is 100%. The description adds no additional parameter meaning beyond the schema, which is acceptable given no parameters exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all active test runs connected to the daemon, providing a specific verb and resource. It distinguishes from siblings like claim_next_batch and get_batch which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving active test runs but provides no explicit guidance on when to use this tool versus alternatives or any prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• release_batchA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It states 'Release a claimed batch' but does not explain what release entails (e.g., batch becomes available again, status changes, reversibility, permissions needed). The description is too brief to inform the agent about side effects or preconditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two sentences with no redundant words. It is front-loaded with the primary action and then provides a usage condition, making it efficient for an agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no output schema, few parameters, no annotations), the description covers the minimum information needed for an agent to decide when to use it. However, it lacks details about post-release state and side effects, which would be important for a complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with each parameter described in the schema. The description adds no additional meaning to the parameters. Baseline score of 3 is appropriate as the description does not detract but also does not enhance understanding of the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Release a claimed batch without providing mock data.' The resource and condition are specific, and it distinguishes from sibling tool provide_batch_mock_data by specifying the absence of mock data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use this tool: 'Use this if you cannot generate appropriate mocks.' It implies the alternative of provide_batch_mock_data for when mocks can be generated, though it doesn't explicitly state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• provide_batch_mock_dataA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must cover behavioral aspects. It does not disclose side effects, error conditions, or state changes. The only behavioral constraint mentioned is the required matching of mocks to requests. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise, consisting of two short sentences and a clear prerequisite note. It efficiently conveys essential information without unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers prerequisites and parameter alignment, but lacks information about return values, error handling, or idempotency. Given no output schema, this leaves gaps in understanding the tool's full behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds value by explaining that batchId and claimToken come from claim_next_batch and that mocks must have exactly one entry per request. This clarifies parameter context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: providing mock response data for a claimed batch. It distinguishes from sibling tools by specifying the prerequisite (claim_next_batch) and the requirement that mocks match the batch requests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs the agent to call claim_next_batch first to obtain batchId and claimToken. Also states that the mocks array must contain exactly one mock per request. This provides clear usage context, though it does not explicitly mention when not to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statusA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It indicates a read operation and specifies what data is included, but does not disclose potential side effects, authorization needs, or rate limits. The behavior is straightforward, so a score of 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear, front-loaded sentence with no wasted words. It efficiently conveys the tool's purpose and scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool with no parameters and no output schema, the description adequately specifies what information is returned. It is complete enough for an agent to understand the tool's function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters and 100% schema coverage, the baseline is 4. The description adds no parameter information because none is needed, but it correctly avoids redundancy.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'current status of mock-mcp daemon', and specifies the contents (active test runs and pending batches). This distinguishes it from sibling tools that deal with batches and runs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking daemon status but lacks explicit guidance on when to use this tool versus alternatives, or when not to use it. No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_batchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses the read-only and non-claiming behavior, which is transparent about side effects. No additional traits (e.g., rate limits) are mentioned, but for a simple read operation, this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently conveys the tool's purpose and safety attributes. No filler, perfectly front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter, read-only tool with no output schema, the description sufficiently covers the essential aspects. It could mention the return format, but the lack is not critical for this straightforward retrieval.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage with a description for batchId. The description adds 'by ID', which is redundant with the schema. Baseline score of 3 is appropriate as no additional parameter semantics are added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves details of a batch by ID. It explicitly adds 'read-only, does not claim', effectively distinguishing it from siblings like claim_next_batch and release_batch.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context that the tool is read-only and non-claiming, guiding use away from mutation operations. While it doesn't explicitly state when to use it, the inference is straightforward from the description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• claim_next_batchA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the batch is locked for a configurable lease (30s default) and that failure to provide data releases the batch for others. This explains critical side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the core purpose. Each sentence adds essential information: purpose, ordering dependency, and lease behavior. No superfluous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the claim operation, lease, and workflow dependency. However, it does not mention the return value (likely the batch data) or error conditions (e.g., no pending batch). With no output schema, this omission slightly reduces completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%; both parameters (runId, leaseMs) are already described in the schema with identical semantics. The description reinforces the leaseMs default but adds no new information beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Claim the next pending mock batch for processing.' This distinguishes it from sibling tools like get_batch (reading) and release_batch (releasing). The verb 'claim' and resource 'mock batch' are specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'You MUST call this before provide_batch_mock_data', providing a clear ordering requirement. It also explains the lease mechanism and timeout behavior, guiding when to use this tool in the workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.