Server Quality Checklist

Tool Scores

• process_logsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It mentions 'efficiently' and 'pagination,' but lacks critical details like performance expectations, error handling, file size limits, or output format. For a tool with 6 parameters and no annotations, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise—two sentences with zero waste. It front-loads the core purpose and includes a practical usage tip. Every word serves a clear function, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, no output schema, no annotations), the description is incomplete. It lacks details on return values, error conditions, performance traits, and how it differs from siblings. While concise, it doesn't provide enough context for an agent to use it confidently without trial and error.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are well-documented in the schema. The description adds minimal value beyond this, only explicitly mentioning 'offset' for pagination and implying pattern matching. It doesn't clarify interactions between parameters or provide examples, aligning with the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Process log files efficiently with pattern matching and pagination.' This specifies the verb ('process'), resource ('log files'), and key capabilities. However, it doesn't explicitly differentiate from sibling tools like 'process_csv' or 'batch_process_csv' beyond the log file focus, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal usage guidance. It mentions using 'offset to skip previous matches,' which hints at pagination scenarios, but offers no explicit when-to-use advice, alternatives, or exclusions compared to siblings. Without context on when to choose this over similar tools, guidance is insufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_toolsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions '95% token savings vs loading all tool definitions' which hints at performance characteristics, but doesn't describe what the search returns (e.g., tool names, descriptions, metadata), whether results are paginated, authentication requirements, rate limits, or error conditions. The behavioral context is insufficient for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences. The first sentence clearly states the core functionality, and the second adds useful performance context. However, the token savings claim feels somewhat promotional and could be integrated more naturally with the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with no annotations and no output schema, the description is incomplete. It doesn't explain what the search returns (e.g., tool metadata, IDs, descriptions), how results are structured, or any limitations. The token savings mention is helpful but doesn't compensate for missing behavioral and output context that an agent needs to use this tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description mentions 'keyword with optional category filter' which aligns with the schema but adds no additional semantic context beyond what's in the parameter descriptions. The token savings mention doesn't relate to parameter usage. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as searching tools by keyword with optional category filtering, using specific verbs ('search') and resources ('tools'). It distinguishes from some siblings like 'batch_process_csv' or 'execute_code' by focusing on discovery rather than execution, though it doesn't explicitly differentiate from 'list_token_efficient_tools' which is also a discovery tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions token savings, but doesn't specify when to prefer this over sibling tools like 'list_token_efficient_tools' or other discovery methods. There are no explicit when/when-not instructions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• batch_process_csvB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions performance benefits ('80% token savings'), it doesn't disclose important behavioral traits like whether this is a read-only or destructive operation, what permissions are required, error handling behavior, rate limits, or what happens when files have different schemas. For a tool that processes multiple files with filtering and aggregation, this is a significant gap in behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded with the core functionality in the first sentence. The second sentence adds valuable performance context. Both sentences earn their place, and there's no wasted verbiage or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters, no annotations, and no output schema, the description is insufficiently complete. It doesn't explain what the tool returns (aggregated results? individual file results? error handling?), doesn't mention the 5-file limit documented in the schema, and provides minimal behavioral context. The performance benefit mention is helpful but doesn't compensate for the missing operational details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so all parameters are well-documented in the schema itself. The description doesn't add any meaningful parameter semantics beyond what's already in the schema - it doesn't explain how parameters interact, provide examples of filter expressions, or clarify the 'aggregate' parameter's effect. With complete schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Process multiple CSV files in a single call with consistent filtering.' It specifies the verb ('process'), resource ('multiple CSV files'), and scope ('single call with consistent filtering'). However, it doesn't explicitly distinguish this tool from its sibling 'process_csv' tool, which appears to be a single-file version.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage guidance by mentioning '80% token savings for multiple files vs individual calls,' which implies this tool should be used when processing multiple CSV files rather than making individual calls. However, it doesn't explicitly state when NOT to use this tool or provide clear alternatives (like when to use 'process_csv' instead). The guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_token_savings_reportB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves a report and best practices, implying a read-only operation, but doesn't clarify aspects like authentication needs, rate limits, data freshness, or whether it's a summary or detailed analysis. For a tool with zero annotation coverage, this leaves key behavioral traits unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any fluff or redundancy. It's front-loaded with the core action and resource, making it easy to parse quickly. Every word earns its place, achieving optimal conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, 100% schema coverage, and no output schema, the description is minimally adequate. It explains what the tool does but lacks details on output format, behavioral constraints, or usage context. For a report-fetching tool with no annotations, it should ideally provide more guidance on what the report contains or how to interpret it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so there are no parameters to document. The description doesn't need to add parameter semantics, and it appropriately avoids mentioning any. A baseline score of 4 is applied as it handles the zero-parameter case correctly without unnecessary details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('detailed token savings report and optimization best practices'), making it immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'list_token_efficient_tools' or 'search_tools', which might also relate to token optimization, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention context, prerequisites, or exclusions, leaving the agent to infer usage based on the name alone. This lack of explicit direction is a significant gap in helping the agent select the right tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_token_efficient_toolsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions 'progressive disclosure' but doesn't explain what each disclosure level entails, whether the operation is read-only, if there are rate limits, or what format the output takes. The behavioral characteristics are underspecified for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise - a single sentence that efficiently communicates the core functionality and parameter options. Every word earns its place with zero waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter with enum), no annotations, and no output schema, the description provides basic but incomplete context. It covers the progressive disclosure concept but lacks details about output format, pagination, or error conditions that would be helpful for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It explains the 'level' parameter's purpose (progressive disclosure) and lists the three enum values, adding meaningful context beyond the bare schema. However, it doesn't detail what each disclosure level specifically returns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: listing available token-efficient tools with progressive disclosure options. It specifies the verb 'list' and resource 'token-efficient tools', though it doesn't explicitly differentiate from sibling tools like 'search_tools' or 'get_token_savings_report'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'search_tools' or 'get_token_savings_report', nor does it specify appropriate contexts or exclusions for using this listing function.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_codeB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'sandboxed environment' (safety context) and '98%+ token savings' (performance trait), but lacks critical details like authentication needs, rate limits, error handling, or what 'sandboxed' entails (e.g., isolation level, resource constraints). For a mutation tool (code execution) with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with three sentences, front-loaded with core functionality. The second sentence adds useful language support, and the third provides performance context, but the token savings claim could be trimmed as it doesn't directly aid tool selection. Overall efficient with minimal waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a code execution tool with 4 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on return values (e.g., stdout/stderr format), error cases, security implications of sandboxing, and prerequisites. The 98% token savings note is tangential rather than essential for tool understanding, leaving gaps in operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75%, providing good baseline documentation for parameters. The description adds value by explicitly listing supported languages (Python, Bash, Node.js) and detailing heredoc syntax for bash, which clarifies the 'code' parameter beyond the schema's generic description. However, it doesn't explain other parameters like 'timeout' or 'response_format' beyond what the schema already covers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Execute code') and resource ('in sandboxed environment'), with explicit mention of supported languages (Python, Bash, Node.js). It distinguishes from sibling tools like batch_process_csv or process_logs by focusing on code execution rather than data processing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for executing code in supported languages, but provides no explicit guidance on when to use this tool versus alternatives like batch_process_csv for data tasks. It mentions heredoc support for bash, which hints at usage for multi-line scripts, but lacks clear when/when-not scenarios or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• process_csvA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions efficiency, token savings, and pagination for large files, which adds useful context beyond basic functionality. However, it lacks details on error handling, performance limits, or what happens with invalid inputs, leaving gaps for a tool with 8 parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with key functionality and includes a specific efficiency tip in just two sentences. Every sentence earns its place by conveying essential information without waste, making it appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (8 parameters, no annotations, no output schema), the description is adequate but has clear gaps. It covers purpose and some usage guidelines but lacks details on return values, error cases, or integration with sibling tools, which would help an agent use it more effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value by implying the tool uses filters, groupby, and pagination, but doesn't provide additional syntax or usage details beyond what's in the schema. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool processes CSV files with specific operations (filters, groupby aggregation, pagination), which is a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'batch_process_csv' or 'process_logs', which might handle similar file processing tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (efficient processing with specific features) and includes a specific guideline for large files (>10K rows) to use offset for token savings. However, it doesn't explicitly mention when not to use it or name alternatives among siblings, such as when to choose 'batch_process_csv' instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.