Server Quality Checklist

Tool Scores

• list_job_runsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool lists runs but doesn't describe key behaviors such as pagination, sorting, filtering (e.g., by status or date), rate limits, authentication requirements, or error handling. This leaves significant gaps in understanding how the tool operates beyond its basic purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured, with a clear purpose statement followed by a brief parameter explanation. It avoids unnecessary words and is front-loaded with the main action. However, the 'Args:' section is slightly redundant since the parameter is already implied, and more critical details (like behavioral traits) are omitted, preventing a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter) and the presence of an output schema (which likely describes return values), the description is minimally complete. It covers the basic purpose and parameter but lacks behavioral context, usage guidelines, and error handling information. This makes it adequate for simple use cases but insufficient for robust agent operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal semantic context beyond the input schema. It explains that 'job_id' is 'The ID of the job to list runs for,' which clarifies the parameter's role but doesn't provide format details, validation rules, or examples. With 0% schema description coverage and only one parameter, this is adequate but not informative, meeting the baseline for such a simple case.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Lists runs') and the target resource ('for a specific job'), making the purpose immediately understandable. It distinguishes this from sibling tools like 'create_job' or 'trigger_run' by focusing on retrieval rather than creation or execution. However, it doesn't specify the scope (e.g., all runs, recent runs, or filtered runs), which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing job ID), exclusions, or comparisons with other tools like 'trigger_run' for initiating runs. The agent must infer usage from the tool name and description alone, which is insufficient for optimal selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_jobB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states it 'Creates a Databricks job' and returns an ID. It doesn't disclose behavioral traits like required permissions, whether it's idempotent, error handling, or side effects, leaving significant gaps for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with a clear purpose statement followed by structured Arg/Return sections. Every sentence adds value, but the 'Args' and 'Returns' labels are slightly redundant with schema fields, keeping it from a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (mutation with 3 params), no annotations, and an output schema (implied by 'Returns'), the description is minimally adequate. It covers basic purpose and parameters but lacks behavioral context and usage guidelines, making it incomplete for safe agent operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful semantics beyond the schema: it explains that 'job_name' is for naming the job, 'package_name' refers to a Python package, and 'remote_wheel_path' is a path to an uploaded wheel file. This compensates well for the 0% schema description coverage, though it could detail format constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Creates a Databricks job') and specifies the key resources involved ('with the specified wheel and entry point'), making the purpose unambiguous. However, it doesn't explicitly differentiate from sibling tools like 'trigger_run' or 'upload_wheel', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'trigger_run' or 'upload_wheel', nor does it mention prerequisites (e.g., needing to upload a wheel first). It implies usage through the action but lacks explicit context or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trigger_runB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the action ('Triggers a run') and return value, but lacks behavioral details such as permissions required, whether the run is synchronous/asynchronous, error handling, rate limits, or side effects. For a mutation tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by clear sections for Args and Returns. Every sentence earns its place by defining parameters and output without redundancy. It's appropriately sized for a tool with two parameters and an output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a mutation tool with no annotations, 0% schema coverage, but an output schema (which handles return values), the description is partially complete. It covers purpose and parameters adequately but lacks behavioral context (e.g., execution model, errors). The output schema reduces the burden, but more guidance on usage and transparency would improve completeness for this complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining 'job_id' as 'The ID of the job to run' and 'job_args' as 'A list of Python parameters to pass to the run', which clarifies their roles beyond schema types. However, it doesn't detail format constraints (e.g., job_id source, job_args syntax), leaving some ambiguity. Baseline 3 is appropriate as it adds value but not fully comprehensive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Triggers') and resource ('run of the specified job'). It distinguishes from siblings like 'create_job' (creation vs. execution) and 'list_job_runs' (listing vs. triggering), though it doesn't explicitly mention these distinctions. The purpose is unambiguous but could be more explicit about sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., job must exist), exclusions, or comparisons to siblings like 'list_job_runs' for monitoring runs. Usage is implied by the purpose but lacks explicit context or decision criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• upload_wheelB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the upload action and return value but omits critical details like required permissions, file size limits, error handling, or whether the operation is idempotent. For a mutation tool, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the core purpose stated first, followed by structured Args and Returns sections. Every sentence earns its place, though the formatting is slightly verbose for a single parameter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a mutation with no annotations) and the presence of an output schema (which covers return values), the description is partially complete. It explains the upload action and parameter but lacks behavioral context like side effects or error conditions, making it adequate but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It adds meaning by explaining that 'local_path' refers to 'The local path to the wheel file', clarifying the parameter's purpose beyond the schema's basic type. However, it doesn't detail format constraints or examples, leaving some ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Uploads') and resource ('a local wheel file to the Databricks workspace'), distinguishing it from sibling tools like 'build_wheel' (creation) or 'list_job_runs' (querying). It precisely defines what the tool does without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'build_wheel' or 'create_job', nor does it mention prerequisites such as needing an existing wheel file. It lacks explicit usage context or exclusions, relying solely on the tool's name and basic purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• build_wheelA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the command executed ('uv build --wheel') and the return value, but doesn't disclose critical behavioral traits like whether this is a read-only operation, what happens on failure, whether it modifies the filesystem, or any side effects. The description is insufficient for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise with zero wasted words. It uses a clear main sentence followed by organized Args and Returns sections. Every sentence earns its place by providing essential information about the tool's purpose, parameters, and return value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that this is a build tool with no annotations but with an output schema (which handles return value documentation), the description is minimally adequate. It covers the basic purpose and parameter semantics but lacks important behavioral context about what the tool actually does beyond the command execution. For a tool that likely modifies the filesystem, more disclosure would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context about the single parameter beyond what the schema provides. While schema description coverage is 0%, the description clarifies that 'target' is 'The path to the directory containing pyproject.toml', which provides essential semantic understanding not present in the schema's minimal title 'Target'. This compensates well for the schema's lack of documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Builds the Python wheel') and the implementation method ('using "uv build --wheel"'), distinguishing it from sibling tools like upload_wheel. It provides a precise verb+resource combination that leaves no ambiguity about the tool's function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like upload_wheel or trigger_run. It mentions the target parameter but doesn't explain prerequisites (e.g., needing pyproject.toml present) or when this operation is appropriate versus other build methods.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.