Server Quality Checklist

Tool Scores

• list_packagesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only mentions that it lists packages with a default parameter, but does not disclose any behavioral traits such as side effects, permissions, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short and front-loaded, covering the main point in one sentence. However, it includes an unnecessary 'Returns' section that adds no useful detail, and could be more succinct.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of output schema and annotations, and the presence of sibling tools, the description is minimally complete but leaves out important context like valid project names or behavior for non-existent projects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description specifies the meaning of the single parameter 'project_name' and its default value, but this largely repeats the input schema. With 0% schema description coverage, it adds minimal value beyond restating schema properties.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'packages' within a project context. However, it does not distinguish from sibling tools like list_models or list_projects, which could cause confusion.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, nor any conditions for usage or prerequisites. The description only states its basic function without usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_modelA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description discloses return type (CompiledModel) and defaults for parameters, offering useful behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise but uses a formal Args/Returns structure that adds length; could be more direct without losing clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description names the return type but does not explain what a CompiledModel contains, leaving some context gaps for a 3-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description lists parameter names and defaults but lacks explanation of valid values, formats, or examples (e.g., what is a valid model_path).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it gets details for a specific model using a verb+resource pattern. However, it does not differentiate from sibling list_models, which might also return details.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when you know the project, package, and model path, but does not provide explicit when-to-use or alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_projectsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It states the return type (list of Project) and that it requires no inputs, but does not explicitly disclose that it is a read-only operation or if there are any side effects. The behavior is adequately implied for a list tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two clear sentences and a return type specification. It front-loads the purpose and wastes no words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (no parameters, no output schema), the description adequately covers the purpose and return type. However, it could mention any limitations (e.g., pagination, filtering) or clarify that it lists all accessible projects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and schema coverage is 100%, so the baseline is 3. The description adds no parameter-specific information because none is needed, but it does not enhance understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'available projects', which is distinct from sibling tools that handle queries, models, and packages. No ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes that no input parameters are required, but it does not provide any guidance on when to use this tool versus alternatives, nor does it mention any prerequisites or context for use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full burden. It discloses that the tool lists models and returns a list, but does not mention read-only nature, authentication requirements, or side effects. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two short sentences plus parameter docs. Front-loaded with the main purpose, no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description specifies return type (List[Model]). It covers the basic function well but omits details like error behavior or handling of missing package_name. Still mostly complete for a simple list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description adds descriptive text for both parameters (e.g., 'The name of the project, defaults to "home"'). This adds meaning beyond the schema's defaults, though details remain minimal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List models for a package,' with a specific verb and resource. It distinguishes from sibling tools like get_model (single model) and list_packages/list_projects (different entities).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for listing models but lacks explicit guidance on when to use this tool versus alternatives like get_model. No exclusion criteria or context cues are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_malloy_queryA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the API endpoint, parameters, return structure, and error codes, but does not specify whether the tool is read-only or has side effects. The examples suggest it is read-only (aggregate query), but this is not explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with Args, Returns, Raises, and Examples sections. It is front-loaded with the core purpose. However, it is somewhat verbose, repeating default values already present in the schema. The examples are helpful but add length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters and no output schema, the description covers the API connection, return structure, error codes, and provides examples. It misses details on authentication, idempotency, and whether queries can modify data. Overall, it addresses most of the query execution context adequately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates fully. It provides detailed parameter explanations, including defaults (e.g., project_name defaults to 'home'), mutual exclusivity between query and query_name, and requirements like source_name being required when using query_name. This adds significant value beyond the schema's titles and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute a Malloy query' and explains it connects to the Malloy Publisher API. It distinguishes from siblings like get_model and list_models, which are about retrieving or listing resources, not executing queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use query vs query_name (mutually exclusive) and provides examples. However, it does not explicitly state when not to use this tool or compare it to alternatives like get_model for exploration.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.