Server Quality Checklist

Tool Scores

• get_project_contextC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. The description mentions 'type inference' but doesn't specify what the tool returns (e.g., a summary, structured data, or raw output), whether it's read-only or has side effects, or any performance considerations. It lacks details on behavior beyond the basic purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized for a simple tool with one parameter, though it could be slightly more informative without losing conciseness. The structure is front-loaded with the core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what the tool returns (e.g., a data structure, success status, or error messages), which is critical for a tool focused on 'type information'. For a tool with no structured output documentation, the description should compensate more by detailing expected results or usage context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the parameter 'project_path' clearly documented as 'Path to the project directory'. The description adds no additional meaning beyond this, as it doesn't elaborate on parameter usage or constraints. With high schema coverage, the baseline score of 3 is appropriate since the schema handles the parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Get project-wide type information for better type inference', which provides a clear verb ('Get') and resource ('project-wide type information'). However, it doesn't specifically distinguish this from sibling tools like 'analyze_python_file' or 'type_check_file' that also deal with type-related operations. The purpose is understandable but lacks sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when this tool is appropriate (e.g., for initial project setup, batch analysis) or when to prefer sibling tools like 'analyze_python_file' for file-specific analysis. There's no explicit or implied context for usage decisions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_types_to_fileC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states the basic action without behavioral details. It doesn't disclose potential side effects (e.g., file modification, backup creation implied by parameter), error conditions, or performance considerations. The mention of 'Pyrefly' hints at external dependency but lacks specifics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded with the core action and resource, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete for a tool that modifies files. It lacks information on success/failure outcomes, error handling, and the impact of using 'Pyrefly'. For a mutation tool with 4 parameters, more context is needed to ensure safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are well-documented in the schema. The description adds no additional parameter semantics beyond implying file modification. Baseline 3 is appropriate as the schema handles parameter details effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Add type annotations') and target resource ('to a Python file'), specifying the tool 'Pyrefly' is used. It distinguishes from siblings like 'analyze_python_file' or 'type_check_file' by focusing on annotation addition rather than analysis or checking. However, it doesn't explicitly contrast with these siblings in the description text itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'type_check_file' or 'analyze_python_file'. The description lacks context about prerequisites (e.g., file must exist, Python version compatibility) or typical scenarios (e.g., during refactoring, for static typing).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• type_check_fileC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool runs type checking, implying a read-only analysis, but doesn't specify if it modifies files, requires specific permissions, has rate limits, or what the output format is. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It front-loads the core action and resource, making it easy to scan. Every word earns its place, providing essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (type checking implies potential for detailed output) and lack of annotations and output schema, the description is incomplete. It doesn't explain what the tool returns (e.g., errors, warnings, success status) or behavioral traits like execution time or dependencies. For a tool with no structured output, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema fully documents the 'file_path' parameter. The description adds no additional meaning beyond what the schema provides (e.g., no details on path format, supported file types, or examples). Baseline 3 is appropriate as the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run type checking') and target resource ('on a Python file'), specifying the tool 'Pyrefly'. It distinguishes from siblings like 'add_types_to_file' (which modifies) and 'analyze_python_file' (which may be broader), but doesn't explicitly contrast them. The purpose is specific but lacks explicit sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'analyze_python_file' or 'get_project_context'. The description implies usage for type checking Python files, but offers no context on prerequisites, exclusions, or comparisons to siblings. It's a basic statement without usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_python_fileB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('analyze') but does not describe what the analysis entails (e.g., static analysis, runtime checks), output format, error handling, or performance considerations. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded with the core action and resource, making it easy to understand at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete for a tool that performs analysis. It does not explain what the analysis returns (e.g., a list of missing annotations, statistics, or recommendations), how results are structured, or any behavioral traits like error conditions. This leaves the agent with insufficient context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for both parameters ('file_path' and 'detailed'). The description does not add any additional meaning beyond what the schema provides, such as explaining how 'detailed' affects the analysis output. Baseline 3 is appropriate as the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb ('analyze') and resource ('Python file') with a precise purpose ('for missing type annotations'). It effectively distinguishes from sibling tools like 'add_types_to_file' (which modifies files), 'get_project_context' (which retrieves context), and 'type_check_file' (which checks types rather than analyzing for missing ones).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying the analysis target ('Python file for missing type annotations'), but it does not explicitly state when to use this tool versus alternatives like 'type_check_file' or 'add_types_to_file'. No guidance is provided on prerequisites, exclusions, or specific scenarios for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.