Server Quality Checklist

Tool Scores

• get_schema_infoC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only states the tool's purpose without detailing behavioral traits such as whether it's read-only, if it requires authentication, rate limits, error handling, or what the output format might be. For a tool with no annotations, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that efficiently conveys the tool's purpose without unnecessary words. It is front-loaded with the core action and resource, making it easy to understand at a glance. Every part of the sentence earns its place by specifying the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a tool that retrieves schema information, the description is incomplete. There are no annotations to clarify behavior, no output schema to explain return values, and the description lacks details on what 'information' includes (e.g., property types, constraints, examples). This makes it inadequate for an agent to fully understand how to use the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the single parameter 'property' documented as 'Schema property to get information about (e.g., 'mark', 'encoding', 'data', 'transform').' The description does not add any additional meaning beyond this, such as syntax details or usage examples. Given the high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Get information about Vega-Lite schema properties and structure,' specifying both the action ('Get information') and the resource ('Vega-Lite schema properties and structure'). However, it does not explicitly differentiate from sibling tools like 'get_example' or 'search_docs,' which might also relate to Vega-Lite documentation or validation, so it lacks sibling differentiation for a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention sibling tools like 'get_example' or 'search_docs,' nor does it specify contexts, prerequisites, or exclusions for usage. This leaves the agent without clear direction on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool searches documentation but doesn't describe how results are returned (e.g., format, pagination), potential limitations (e.g., search scope, rate limits), or error handling. For a search tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is appropriately sized for a simple tool, though it could be slightly more structured by front-loading key details like the resource name earlier.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a search function with one parameter) and lack of annotations and output schema, the description is incomplete. It doesn't explain what the search returns (e.g., links, snippets, full text), how results are formatted, or any behavioral traits like search limitations. For a tool with no structured output information, more context is needed to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'query' parameter documented as 'Search query (e.g., 'bar chart', 'color encoding', 'scale domain')'. The description adds minimal value beyond this, only implying the query is for Vega-Lite topics. Since schema coverage is high, the baseline score of 3 is appropriate, as the description doesn't significantly enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search through Vega-Lite documentation for information about charts, encodings, marks, and more.' It specifies the verb ('Search') and resource ('Vega-Lite documentation'), and mentions the types of content covered. However, it doesn't explicitly differentiate from sibling tools like 'get_example' or 'get_schema_info', which might also retrieve documentation-related information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose 'search_docs' over sibling tools such as 'get_example' (which might fetch specific examples) or 'get_schema_info' (which could provide schema details), nor does it specify any prerequisites or exclusions for its use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_specC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool validates and checks for errors, but doesn't disclose behavioral traits like what types of errors are detected, whether validation is strict or lenient, if it returns detailed error messages, or any performance considerations. This leaves significant gaps for a validation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It's front-loaded with the core purpose, making it easy to understand quickly. Every part of the sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (validation of nested JSON specs) and lack of annotations and output schema, the description is insufficient. It doesn't explain what the validation entails, what outputs to expect (e.g., success/failure, error details), or how errors are reported. For a validation tool with no structured output, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'spec' parameter documented as 'Vega-Lite JSON specification to validate'. The description adds no additional meaning beyond this, such as format examples or validation scope. With high schema coverage, a baseline score of 3 is appropriate as the schema handles the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('validate') and resource ('Vega-Lite specification'), and mentions checking for errors. It doesn't explicitly differentiate from sibling tools like 'get_example' or 'search_docs', but the validation focus is distinct enough to avoid confusion.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing a valid spec, or compare it to siblings like 'get_schema_info' for schema-related tasks. Usage is implied by the name but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_exampleB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states the basic operation. It doesn't disclose important behavioral traits like whether this is a read-only operation, if it requires authentication, rate limits, what format the examples are returned in, or how many examples might be retrieved. The description is minimal and lacks operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's appropriately sized for a simple retrieval tool and front-loads the core purpose immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter retrieval tool with no output schema, the description is minimally complete but lacks important context. It doesn't explain what format the examples are returned in, whether this is a filtered list or single example, or how results are structured. The absence of annotations means more behavioral disclosure would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions parameters ('by category or type') but doesn't add meaningful semantics beyond what the 100% schema coverage already provides. The schema fully documents both parameters with clear descriptions and examples, so the description's minimal parameter mention doesn't add value, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Retrieve') and resource ('Vega-Lite example specifications'), making the purpose unambiguous. However, it doesn't differentiate this tool from its siblings (get_schema_info, search_docs, validate_spec) which handle different types of Vega-Lite-related operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings. It doesn't mention alternatives like search_docs for documentation or validate_spec for validation, nor does it specify when this retrieval tool is appropriate versus other options.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.