Server Quality Checklist

Tool Scores

• create_test_caseB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It states 'Create ... in the database' which implies persistence, but it does not disclose whether the operation is idempotent, whether the company must already exist, what happens on duplicates, or any error behavior. Minimal behavioral detail beyond the obvious create action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It efficiently communicates the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (6 parameters, nested objects, 4 required), the description is too sparse. It does not explain when to use this tool, what it returns, or any constraints beyond 'for a specific company'. The schema covers parameters, but the description fails to provide contextual completeness for a moderately complex tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema descriptions cover 100% of the parameters, so the baseline is 3. The description adds nothing beyond schema: 'for a specific company' merely echoes the existing company_name parameter's schema description. No additional semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Create') and identifies the resource ('test case') and context ('for a specific company'). It clearly distinguishes this creation tool from sibling tools like run_tests and list_test_cases, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance on when to use this tool versus alternatives. It merely states the action, leaving the agent to infer usage from the existence of sibling tools. There are no exclusions or alternative references.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_draft_promptsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It implies a read-only list operation but does not mention return format, pagination, or any limitations beyond company scope. The phrase 'all saved draft prompts' adds some context but largely just restates the tool's name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, eight-word sentence that immediately states the action and resource. There is no redundancy or extraneous detail, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one well-documented parameter, the description conveys the core purpose adequately. However, without an output schema or annotations, the agent is left without details on response structure or behavior, and the description does not address potential limitations or alternatives.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides a thorough description of company_name, including format and example, and schema coverage is 100%. The tool description's mention of 'a specific company' aligns with the parameter but adds no new semantic value, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (List), the resource (saved draft prompts), and the scope (for a specific company). This distinguishes it from sibling tools like list_test_cases and get_draft_prompt, which target different resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention that get_draft_prompt is for retrieving a single prompt or that save_draft_prompt is for creating them, leaving the agent without clear selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_toolA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It indicates a read-only operation ('Get') but does not disclose the role of company_name (likely scoping), error behavior, or return format, leaving notable gaps for a simple get tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no unnecessary words. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, but the description does not explain why company_name is required or what 'details' entails. With no output schema, the agent lacks clarity on return values. It is a minimum viable description but leaves some contextual gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover 100% of parameters, so the baseline is 3. The description only echoes 'by ID', aligning with tool_id, and provides no extra meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('details of a specific tool') with a specific identifier ('by ID'), effectively distinguishing it from list_tools which lists all tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a specific tool ID is known, but it does not explicitly mention alternatives or when not to use this tool. It provides clear context but lacks explicit exclusions or comparisons to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• save_improvement_runA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action ('Save') but does not reveal whether it creates a new record or overwrites an existing one, whether the company must already exist, or any other side effects. This is a significant transparency gap for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that immediately states the action and the object. It contains no filler or redundant information, and every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a complex input schema (nested objects, many required fields), but the schema itself is comprehensive and fully documents the structure. The description supplies the overall purpose while the schema covers the parameters. The main gap is the lack of any mention of return value or post-save behavior, but for a save operation the purpose is clear enough to be considered mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with detailed descriptions for every parameter, including examples (e.g., company_name in kebab-case). The description adds only a high-level grouping of parameters ('prompts, analysis, and test results') without providing syntax or format details beyond what the schema already specifies, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Save') and resource ('complete improvement cycle'), and clarifies the contents ('prompts, analysis, and test results') and scope ('for a specific company'). This clearly distinguishes it from sibling tools like save_draft_prompt, which only saves a single prompt.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is implied: it is used after an improvement cycle has been completed. However, there is no explicit guidance on when to prefer this over related tools (e.g., save_draft_prompt) or when not to use it, so it lacks the exclusionary detail that would merit a higher score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_toolsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It says 'List all available tools' but gives no details about response format, security, pagination, or what 'available' means. This is minimal and not much beyond the tool name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is just two short sentences, front-loaded with the core purpose. No wasted words—every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With only one parameter, no annotations, and no output schema, the description covers the basic purpose and a use case. However, it does not specify what the response contains (e.g., tool names, schemas), leaving some ambiguity for the agent. It is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the sole parameter company_name, including a clear example. The description adds no extra parameter information, but the schema already handles it fully, so a baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all available tools for a company, providing a specific verb, resource, and scope. It distinguishes from sibling tools like list_test_cases and list_skills by focusing on the 'tools' resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear usage context: 'Use this when generating tests to know what tool calls are possible.' However, it does not mention alternatives or when not to use it, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_skillsA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It only states that it lists skills, implying a read-only operation, but does not mention side effects, permissions, return format, or limitations. This is minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no unnecessary words. It efficiently conveys the tool's purpose without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless listing tool, the description covers the core functionality. However, given sibling tools like list_tools and the absence of an output schema or annotations, a brief note on how skills differ from tools or what the return data looks like would improve completeness. Still, the low complexity makes this largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the input schema fully covers parameter semantics (100% coverage). The description adds no parameter details, but with no parameters, this is not a gap. Baseline of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'List all available skills' uses a specific verb ('list') and resource ('skills'), clearly stating the action and scope. It distinguishes from sibling tools like list_tools and get_tool by targeting skills rather than tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (when you need to enumerate skills) but does not explicitly state when to use this tool versus alternatives like list_tools, nor any exclusions. It provides no guidance on selecting between related tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_test_casesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. It indicates a read-only operation via 'List' but does not disclose return format, pagination, ordering, or permission requirements. It provides basic transparency but lacks deeper behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence of 12 words with no redundant information. It is front-loaded and every word contributes meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one well-documented parameter and no output schema, the description sufficiently conveys the purpose and scope. It covers the essential context, though it could optionally mention the return structure or any limits, but this is not critical for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already fully describes the only parameter 'company_name' with a clear description and example. The tool description adds nothing beyond what the schema provides, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List'), the target resource ('test cases'), and the scope ('for a specific company'). It distinguishes this from sibling tools like create_test_case (creating) and run_tests (running), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used to retrieve test cases for a particular company, but it does not explicitly state when to use it instead of other list tools (e.g., list_skills, list_draft_prompts) or mention any exclusions. Usage context is present but not fully explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testsA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'real LLM evaluation' implying cost/latency, but does not disclose side effects (e.g., whether results are saved, if a run record is created), rate limits, or the response/output format. For a tool that executes tests, this is a significant transparency gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences. The first sentence front-loads the core purpose ('Run regression tests against a system prompt with real LLM evaluation'), and the second covers the crucial parameter decision. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a 6-parameter tool with full schema descriptions, but lacks information about return values or output format (no output schema), and does not explain the role of other parameters like tools, test_model, or test_case_ids beyond the schema. Given the tool's potential complexity (running LLM evaluations), the description should mention what the caller receives or what side effects occur.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds meaningful semantic guidance by explaining the relationship between prompt_id and system_prompt, noting that prompt_id is 'from save_draft_prompt' and preferred for 'faster performance'. This goes beyond the schema's simple 'use one or the other' instruction.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description immediately states the specific action 'Run regression tests' against a 'system prompt' with 'real LLM evaluation' for a specific company. This clearly distinguishes it from sibling tools like create_test_case or list_test_cases, which create or list rather than execute tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on parameter usage: 'Use prompt_id (from save_draft_prompt) for faster performance, or system_prompt for direct text.' This is practical guidance for when to use each option. However, it does not explicitly exclude alternatives or contrast with other tools, though no sibling tool performs the same test-running function.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_draft_promptA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the burden of disclosing behavioral traits. It correctly implies a read-only action ('retrieve'), but it does not mention what happens if the prompt is not found, whether it returns null or errors, or any other side-effect-free confirmation. The description adds minimal behavioral context beyond the obvious read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the verb and precisely states the action and scope. Every word contributes to meaning with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple getter tool with two well-documented parameters. The description fully covers the purpose and scope, but it does not mention the return format or error behavior, which would normally be expected given the absence of an output schema. Still, for the tool's simplicity, the description is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description reinforces that prompt_id is the identifier and company_name is the scoping company, but it does not add new details beyond the schema, such as format or validation rules. It adequately aligns with the schema but does not enhance it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('retrieve') and clearly identifies the resource ('previously saved draft prompt') and the scoping ('by ID for a specific company'). This strongly distinguishes it from sibling tools such as list_draft_prompts (listing) and save_draft_prompt (saving).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: use this when you need a specific saved draft prompt by ID for a company. It does not explicitly name alternatives or say when not to use, but the retrieval semantics are unambiguous. A score of 4 reflects the lack of explicit alternative guidance while acknowledging the clarity of the intended use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• save_draft_promptA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It states that the tool saves to a database and returns a prompt_id, but does not disclose potential side effects such as overwriting existing drafts, required permissions, or idempotency behavior. This is a moderate disclosure level for a write operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences deliver the purpose, target, return value, and downstream usage with no redundant words. The structure is front-loaded with the action and resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description appropriately mentions the return value (prompt_id) and how it connects to run_tests. Given its simplicity and the schema's completeness, the description adequately covers the tool's role and expected outcome, though it does not elaborate on handling of optional labels or edge cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of the parameters with descriptions, so the description adds little beyond mentioning the return value. It does not describe parameter formats or constraints beyond what the schema already provides. This matches the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Save') and identifies the exact resource ('draft prompt') plus the target context ('for a specific company'). It also distinguishes this tool from siblings such as get_draft_prompt, list_draft_prompts, and run_tests by explicitly stating its role in saving for later testing and returning a prompt_id for run_tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: to save a draft prompt for later testing, and it connects the returned prompt_id to run_tests. It doesn't explicitly name alternatives or exclusions, but the intended use is unambiguous given the sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.