Server Quality Checklist

Tool Scores

• basedpyrightC
  2.2/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations provided, so the description must carry the full burden of behavioral disclosure. The description offers no information about side effects, file modifications, network access, or permissions required. It simply says 'Run', which is minimally informative and does not disclose any behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence and front-loaded, but it is under-specified rather than genuinely concise. It omits necessary context about parameters and behavior, so the brevity does not serve the agent well.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter, an output schema (per context), and no annotations, the description is incomplete. It does not explain what 'diagnostics' entails, how the output is structured, or how the 'paths' parameter affects behavior, leaving major gaps for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter 'paths' with 0% description coverage, and the tool description does not mention it at all. The description fails to explain what 'paths' means, how it should be provided, or its default behavior, offering no compensation for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Run basedpyright diagnostics' clearly identifies the tool's action ('Run') and resource ('basedpyright diagnostics'). It is specific enough to convey its purpose, but it does not explicitly differentiate from siblings like pytest or ruff_lint, though the name itself implies a type-checking focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives (e.g., pytest or ruff_lint), no prerequisites, and no exclusion criteria. It only states what the tool does, leaving the agent to infer appropriate usage scenarios without support.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ruff_lintC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure, but it only offers the generic phrase 'Run Ruff diagnostics'. It fails to disclose whether the operation is read-only, how output is returned, how the 'paths' parameter affects behavior, or whether any files or settings are modified, leaving significant ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence—'Run Ruff diagnostics'—with no unnecessary words or repetition. It is maximally concise for the limited information it conveys.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers only the bare action and omits usage guidance, parameter semantics, and behavioral details. While an output schema exists and the tool is relatively simple, the lack of contextual information (e.g., when to use it, what 'paths' does) makes the overall description inadequate for confident tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'paths' has no description and the schema coverage is 0%. The description does not compensate by explaining what 'paths' represents, its default behavior, or the expected input format. The agent is left with no semantic understanding of the parameter beyond its name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run') and the resource ('Ruff diagnostics'), identifying the tool as a linter. It is distinguishable from sibling tools basedpyright and pytest by naming Ruff, though it is slightly vague about what 'diagnostics' entails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No usage guidance is provided. The description does not explain when to use this tool instead of basedpyright or pytest, nor does it mention any prerequisites, file types, or configuration context. It simply states what the tool does without any situational guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pytestC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for disclosing behavioral traits. It only mentions that it runs pytest and returns the JUnit XML report, but omits details about side effects, exit code behavior, timeout potential, or any required environment setup, which are important for a test-execution tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no filler. It is appropriately sized and front-loaded, conveying the core action and output format efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although the tool is simple and an output schema exists, the description leaves the `paths` parameter undefined and provides no context about test execution semantics or usage. This sparse description is insufficient for an agent to fully understand the tool's scope and potential side effects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter `paths` has no description in the schema (coverage 0%) and is not mentioned in the tool description. While the name gives a hint, the description does not explain what paths should contain or how they are used, leaving the agent without sufficient parameter context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Run') and resource ('pytest') and specifies the output format ('JUnit XML report'). It does not explicitly distinguish itself from sibling tools like basedpyright or ruff_lint, but its function as a test runner is inherently distinct from those linters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor any context about prerequisites or exclusions. It merely states what the tool does, leaving the agent to infer when it should be used.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.