Server Quality Checklist

Tool Scores

• run_pythonC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'Supports any file format and Python libraries', which hints at capabilities, but lacks critical behavioral details: it doesn't mention security implications (e.g., sandboxing, permissions), resource limits (e.g., execution time, memory), error handling, or output format. For a powerful tool like code execution, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences that efficiently cover the tool's purpose and capabilities. It's front-loaded with the core function ('Execute Python code') and avoids unnecessary details. However, it could be slightly more structured by separating use cases or adding a brief example, but overall, it's well-sized with minimal waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (code execution with 3 parameters, no annotations, but an output schema exists), the description is moderately complete. It covers the high-level purpose and capabilities but lacks depth in usage guidelines, behavioral transparency, and parameter semantics. The presence of an output schema means return values are documented elsewhere, reducing the burden on the description, but for a tool with potential security and resource implications, more context is needed to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (code, packages, file_paths) with descriptions. The description adds no additional meaning beyond what the schema provides—it doesn't explain parameter interactions, provide examples, or clarify semantics like how 'packages' mappings work or how 'file_paths' are accessed. Baseline 3 is appropriate since the schema does the heavy lifting, but the description doesn't compensate with extra insights.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with the verb 'Execute' and resource 'Python code', specifying it's for 'flexible file operations, data processing, and custom tasks'. It distinguishes from sibling tools (get_document_info, read_document, write_document) by focusing on code execution rather than document operations, though it doesn't explicitly name those alternatives. The purpose is specific but could be more precise about what distinguishes it from similar code execution tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions 'flexible file operations, data processing, and custom tasks' but doesn't specify scenarios where run_python is preferred over sibling tools like write_document for file operations or other code execution methods. There's no mention of prerequisites, constraints, or typical use cases, leaving the agent with minimal contextual guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_documentC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It implies a write operation ('Write') but lacks critical behavioral details: it doesn't disclose whether this overwrites existing files, requires specific permissions, handles errors, or has side effects like file creation. For a mutation tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded, using a single phrase that efficiently conveys the core purpose and supported formats. Every word earns its place with no redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a write operation with 3 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on behavioral traits, error handling, file system interactions, and return values, making it inadequate for safe and effective tool invocation by an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema by listing supported formats, but it doesn't provide additional context on parameter usage, constraints, or examples. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Write') and resource ('document content'), specifying the supported formats (Excel, Word, PowerPoint, Text). It distinguishes from sibling tools like 'read_document' by focusing on writing rather than reading. However, it doesn't explicitly differentiate from 'get_document_info' or 'run_python' in terms of document creation vs. metadata retrieval or execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., file system access), when-not-to-use scenarios (e.g., for reading documents), or explicit alternatives among sibling tools like 'read_document' for reading or 'run_python' for other operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_document_infoB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states 'Get document metadata' which implies a read-only operation, but doesn't disclose behavioral traits like error handling (e.g., if file doesn't exist), performance considerations, or authentication needs. The description is minimal and lacks context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded with the core purpose and includes helpful examples (page count, etc.) that add value without verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (simple metadata retrieval), 100% schema coverage, and the presence of an output schema (which handles return values), the description is reasonably complete. It clearly states what the tool does, though it lacks usage guidelines and behavioral details that could enhance agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds no parameter-specific information beyond implying metadata retrieval, which is covered by the tool's purpose. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'document metadata' with specific examples (page count, sheet count, slide count, file size, etc.). It distinguishes from sibling tools like 'read_document' (which likely reads content) and 'write_document' (which modifies content), though it doesn't explicitly mention these distinctions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., file existence), when not to use it, or how it differs from siblings like 'read_document' beyond the metadata focus implied in the purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_documentA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses key behavioral traits: format support, read modes (raw/paginated), and interactive UI for specific formats. However, it doesn't mention performance characteristics, error handling, authentication requirements, or rate limits that would be important for a document reading tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured in two sentences that each add value: first establishes core functionality with format enumeration, second adds mode and UI information. No redundant information, though it could be slightly more front-loaded with the most critical information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, multiple formats and modes) and the presence of an output schema, the description provides good foundational context. It covers what the tool does, supported formats, and key operational modes. The output schema existence means return values don't need explanation, making this reasonably complete for agent usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds some context about format support and modes that complements the schema, but doesn't provide additional parameter semantics beyond what's already documented in the schema descriptions. The description doesn't explain parameter interactions or provide usage examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb ('Read') and resource ('document content'), enumerates supported file formats, and distinguishes from sibling tools like 'get_document_info' (metadata) and 'write_document' (write operation). It provides comprehensive scope information beyond just the tool name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through format support and mode options, but doesn't explicitly state when to use this tool versus alternatives like 'get_document_info' for metadata or 'write_document' for modifications. No explicit when-not-to-use guidance or prerequisite information is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.