Server Quality Checklist

Tool Scores

• submit_audit_resultD
  1.9/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. 'Submit' implies a write operation, but the description fails to disclose critical behavioral traits such as whether this requires specific permissions, what happens after submission (e.g., irreversible changes, notifications), or any rate limits. This leaves the agent with significant uncertainty about the tool's effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise—a single four-word sentence—and front-loaded with the core action. However, this brevity comes at the cost of under-specification; while there is no wasted text, the description fails to provide necessary context that would help the agent use the tool effectively.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a submission tool with three parameters (two required) and no annotations, the description is incomplete. While an output schema exists (which might cover return values), the description lacks essential context about the tool's purpose, usage, behavior, and parameter meanings. This makes it inadequate for safe and effective use by an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning none of the three parameters (passed, issues, score) are documented in the schema. The description adds no parameter semantics beyond what the schema provides—it doesn't explain what 'passed' means, what constitutes an 'issue', how 'score' is used, or the relationship between these parameters. This leaves all parameters effectively undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Submit the audit result' is a tautology that essentially restates the tool name 'submit_audit_result'. It provides no additional specificity about what resource is being submitted, to whom, or what the audit entails. While it does contain a verb ('submit') and a resource ('audit result'), it lacks any distinguishing details that would help differentiate it from potential alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to use this tool versus alternatives like 'submit_draft' or 'update_rules'. There is no mention of prerequisites, appropriate contexts, or exclusions. The agent must infer usage solely from the tool name and parameters, which is insufficient for clear decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_draftC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'submit' implying a write operation, but fails to describe key traits like whether this requires authentication, what happens after submission (e.g., triggers an audit process), or any rate limits. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words, making it easy to parse. It is appropriately sized for a simple tool, though this conciseness comes at the cost of missing details in other dimensions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema, the description doesn't need to explain return values. However, with no annotations, 2 parameters (0% schema coverage), and a mutation implied by 'submit', the description is incomplete—it lacks behavioral context and parameter details, making it only minimally adequate for a basic submission tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for undocumented parameters. It does not explain the 'code' or 'language' parameters beyond what the schema provides (e.g., what format 'code' should be in, what 'language' options exist beyond the default 'python'). This adds minimal value, failing to address the coverage gap adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('submit') and resource ('code draft for audit'), making the purpose understandable. However, it doesn't differentiate this tool from sibling tools like 'submit_audit_result' or 'update_rules', which also involve submission or updating processes in the audit context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'submit_audit_result' or 'update_rules'. It lacks context about prerequisites, such as whether a draft must be prepared first or if this is part of a workflow, leaving usage ambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reset_sessionB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. 'Reset' implies a destructive or state-changing operation, but the description doesn't clarify what gets reset (e.g., data loss, session state), whether authentication is needed, or any side effects. This is inadequate for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste—it directly states the tool's purpose without unnecessary elaboration. It's appropriately sized and front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, 100% schema coverage, and an output schema exists, the description's minimal approach is somewhat adequate. However, as a mutation tool ('reset') with no annotations, it lacks critical behavioral context like effects or permissions, making it incomplete for safe invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the lack of inputs. The description doesn't need to add parameter details, and the baseline for this scenario is 4, as it appropriately avoids redundant information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('reset') and target resource ('current audit session'), providing a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'submit_audit_result' or 'update_rules' which also operate on audit sessions, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'submit_draft' or 'update_rules'. There's no mention of prerequisites, timing considerations, or exclusions, leaving the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_rulesA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool performs operations like 'add', 'remove', 'update', and 'list', implying mutation capabilities. However, it lacks details on behavioral traits such as permission requirements, whether changes are reversible, rate limits, or error handling. The description adds basic context but misses critical information for a mutation tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, args, returns, examples) and uses bullet points for readability. It's appropriately sized, with each sentence earning its place by explaining parameters or providing examples. However, the 'Args' section could be more front-loaded with the purpose statement, and some redundancy exists in parameter explanations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, mutation operations) and no annotations, the description does a good job covering inputs with detailed parameter semantics and examples. Since an output schema exists, it doesn't need to explain return values beyond the brief 'Status message' note. The main gap is lack of behavioral context like permissions or side effects, but overall it's fairly complete for the provided context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate fully. It provides detailed semantics for all 5 parameters: 'action' with operation types, 'rule_id' as identifier with usage contexts, 'severity' with levels, 'description' as rule text, and 'weight' as point deduction range. This adds significant meaning beyond the bare schema, effectively documenting each parameter's purpose and constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Update audit rules configuration' with a specific verb ('update') and resource ('audit rules'). It distinguishes itself from sibling tools like 'submit_audit_result' or 'submit_draft' by focusing on rule management rather than audit submissions. However, it doesn't explicitly differentiate from 'reset_session' which might also affect configurations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the 'action' parameter values ('add', 'remove', 'update', 'list'), suggesting when to use each operation. It provides examples for different scenarios but doesn't explicitly state when to choose this tool over alternatives like 'reset_session' or mention prerequisites such as authentication needs. The guidance is contextual but lacks explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.