Server Quality Checklist

Tool Scores

• ruff_checkC
  2.6/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It provides zero behavioral traits: no error handling, output format description, performance notes, or side effects beyond stating it runs linting.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence is concise but under-specified for a tool with 5 parameters and sibling context. Lacks structure or front-loading of key details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite high schema coverage, the description lacks contextual details such as output behavior, error cases, and usage scenarios relative to siblings. Incomplete for a 5-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, all parameters have clear descriptions. The tool description adds no additional meaning beyond the schema, meeting the baseline expectation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the action ('Run') and resource ('Ruff linting') on files or directories. It distinguishes from sibling tools like ruff_fix or ruff_format by focusing on linting, but does not explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings (ruff_fix, ruff_format). No prerequisites, limitations, or context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ruff_formatC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full responsibility for disclosure. It states 'Format Python code' but does not clarify that files are modified in-place, whether changes are reversible, or if any safety checks are performed. The agent lacks critical behavioral context for a mutation operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with no redundancy. Extremely concise and front-loaded, every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having 4 parameters and no output schema, the description does not explain return values (e.g., what is output after formatting?), side effects (file modification), or error behavior. This is insufficient for a tool that modifies files, especially without annotations to fill gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers all parameters (100% coverage). The description adds minimal value beyond the schema, such as explaining the auto-discovery behavior for config_path. For line_length and check_only, the description essentially repeats schema documentation. Overall, marginal enrichment.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool formats Python code using Ruff. The verb 'Format' and resource 'Python code' are specific, and it distinguishes from sibling tools like ruff_check and ruff_fix by focusing on formatting, though it could explicitly mention that it applies code style.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like ruff_check or ruff_fix. There is no mention of typical use cases, prerequisites, or scenarios where formatting is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pytest_runnerB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only states 'Run pytest tests' without disclosing behavioral traits such as side effects (e.g., no file modification), required dependencies, or how results are returned. The description adds minimal insight beyond the name.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no wasted words. It is front-loaded with the core action and resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 13 optional parameters and no output schema, the description is too minimal. It does not explain how parameters interact (e.g., test_function vs test_class), default behavior when no parameters are provided, or what the output looks like. For a complex testing tool, more detail is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description does not elaborate on parameter meanings beyond what the schema already provides, nor does it explain relationships or typical usage patterns. It adds no extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run pytest tests') and the resource ('specific files, test functions, or entire test suite'), distinguishing it from the sibling ruff_* tools which are for linting and formatting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. While the purpose distinguishes it from siblings, there is no mention of prerequisites, context, or when not to use it (e.g., if pytest is not installed).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ruff_fixB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears full responsibility for disclosing behavioral traits. It mentions 'auto-fix' but does not clarify that this modifies files (destructive behavior), nor does it explain what 'where possible' means (some violations may be unfixable). The 'unsafe' parameter is named but its implications are not described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, very concise. It earns its place by clearly stating the core purpose. However, it could include a bit more context without becoming overly verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 3 parameters and no output schema, yet the description provides no information about return values, behavior (e.g., modifies files in place), or safety considerations. Given the presence of sibling tools, the description does not help the agent decide when to use this tool. Completeness is insufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the input schema already describes all parameters. The description adds no additional meaning beyond the schema. Baseline score of 3 is appropriate as the description does not compensate for any gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Auto-fix linting violations where possible'. It uses a specific verb ('fix') and resource ('linting violations'), and the name 'ruff_fix' together with the description distinguishes it from sibling tools like 'ruff_check' (check only) and 'ruff_format' (formatting).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal usage guidance. It implies the tool is for automatically fixing linting issues, but does not explicitly state when to use this tool over alternatives like 'ruff_check' or 'ruff_format'. No conditions, prerequisites, or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.