Server Quality Checklist

Tool Scores

• create_jobB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only provide a title, so the description carries full burden. It states this creates a new job but lacks behavioral details like required permissions, whether it's idempotent, what happens on failure, or if it triggers immediate execution. The mention of 'minimal set of fields' hints at constraints but is vague.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. It's appropriately sized for a tool with good schema coverage, making every word count.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and minimal annotations, the description is incomplete for a creation tool. It covers the basic action but lacks details on return values, error handling, or behavioral traits, leaving gaps that could hinder agent usage despite good schema coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents parameters. The description adds no additional meaning beyond implying 'minimal fields' might relate to required parameters, but it doesn't clarify which fields are minimal or provide usage context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create') and resource ('Agent Job'), specifying it's a new job with minimal fields. It distinguishes from siblings like 'cancel_job' or 'get_job' by being the creation tool, though it doesn't explicitly contrast with 'list_jobs' or 'get_jobs_stats'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives is provided. It doesn't mention prerequisites, dependencies, or contrast with sibling tools like 'list_jobs' for viewing existing jobs or 'cancel_job' for stopping them, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_jobB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only provide a title, so the description carries the full burden. It states the action is to cancel a job, implying a destructive mutation, but lacks details on permissions needed, whether cancellation is reversible, rate limits, or what happens post-cancellation. It adds basic context but misses key behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core action and resource, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive mutation tool with no annotations beyond a title and no output schema, the description is inadequate. It lacks information on permissions, side effects, error handling, or return values, leaving significant gaps in understanding the tool's behavior and context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear parameter documentation in the schema. The description doesn't add any meaning beyond what the schema provides, such as explaining parameter interactions or usage nuances. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('cancels') and target resource ('an agent job by its ID'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_job' or 'list_jobs' beyond the obvious action difference, missing specific sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives or any prerequisites. While the action 'cancels' implies it's for active jobs, there's no explicit mention of job state requirements, error conditions, or comparison with siblings like 'create_job'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_jobB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds minimal behavioral context beyond what annotations provide. Annotations include a title ('Get Agent Job') but no other hints like readOnly or destructive. The description implies a read operation ('retrieves'), which aligns with typical retrieval behavior, but doesn't disclose details such as error handling, authentication needs, or rate limits. No contradiction with annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded, consisting of a single sentence that directly states the tool's purpose. There is no wasted verbiage or unnecessary elaboration, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (a retrieval operation with 2 parameters and no output schema), the description is minimally adequate. It states what the tool does but lacks context on usage guidelines, behavioral details, or output format. With no annotations beyond title and no output schema, more completeness would be beneficial, but it meets a basic threshold for this low-complexity tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description doesn't add any parameter semantics beyond what the input schema provides. Schema description coverage is 100%, with clear descriptions for both 'job_id' and 'org_id', including examples. The description itself doesn't mention parameters, so it relies entirely on the schema, meeting the baseline score for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('retrieves') and resource ('an agent job by its ID'), making the purpose specific and understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_job_type' or 'list_jobs', which might retrieve similar job-related information but with different scopes or parameters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. For example, it doesn't mention that this is for retrieving a single job by ID, as opposed to 'list_jobs' for multiple jobs or 'get_job_type' for job metadata. There are no explicit when/when-not instructions or prerequisites stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_typeB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide a title ('Get Job Type Configuration') but no explicit hints like readOnlyHint or destructiveHint. The description adds minimal behavioral context by implying a read operation ('Retrieves'), but it doesn't disclose details such as authentication needs, rate limits, or error handling, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It is front-loaded and appropriately sized, making it easy to parse and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (2 parameters, no output schema, simple schema), the description is minimally adequate. It covers the basic purpose but lacks details on output format, error cases, or integration with siblings, making it incomplete for fully informed usage without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters ('job_type_id', 'org_id') well-documented in the schema. The description adds no additional meaning beyond the schema, such as explaining parameter interactions or usage nuances, so it meets the baseline for high coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Retrieves') and resource ('an agent job type by its ID'), making the purpose specific and understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_job' or 'list_jobs', which might retrieve different job-related data, so it misses full sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as 'get_job' or 'list_jobs'. It lacks context on prerequisites, exclusions, or specific scenarios, offering only a basic statement of function without usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_jobsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds minimal behavioral context beyond what's implied by the name and annotations. It mentions 'optional filters and pagination' which provides some operational context, but doesn't disclose important behavioral traits like rate limits, authentication requirements, or what happens when no filters are applied. With no annotations provided, the description carries the full burden but does only the bare minimum.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Retrieves a list of agent jobs') and then adds qualifying information about filters and pagination. Every word serves a purpose with zero redundancy or wasted space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list operation with excellent schema coverage (100%) but no output schema, the description provides the minimum viable context. It states what the tool does but lacks information about return format, pagination behavior details, error conditions, or how it differs from sibling tools. The absence of annotations means more behavioral disclosure would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already provides comprehensive documentation for all 12 parameters. The description adds no additional parameter semantics beyond mentioning that filters and pagination are 'optional' - which is already evident from the schema's lack of required parameters. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Retrieves') and resource ('list of agent jobs'), making the purpose immediately understandable. However, it doesn't differentiate this tool from sibling tools like 'get_job' or 'get_jobs_stats', which would require explicit comparison to achieve a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions 'optional filters and pagination' but provides no guidance on when to use this tool versus alternatives like 'get_job' (for single job details) or 'get_jobs_stats' (for aggregated statistics). There's no mention of prerequisites, typical use cases, or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_jobs_statsA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only provide a title, so the description carries the burden of behavioral disclosure. It adds useful context about optimization for dashboards and minimal network overhead, which hints at performance characteristics. However, it lacks details on rate limits, authentication needs, or specific return format, leaving some behavioral aspects unclear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and followed by usage context. Every word earns its place—no redundancy or fluff. It efficiently communicates key information without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (9 parameters, no output schema, and minimal annotations), the description is adequate but incomplete. It covers purpose and usage well but lacks details on output format, error handling, or behavioral constraints like pagination. For a statistical tool with many filters, more context on result structure would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 9 parameters. The description does not add any parameter-specific information beyond what the schema provides, such as explaining how filters combine or default behaviors. This meets the baseline for high schema coverage but doesn't enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Get aggregated statistics') and resource ('agent jobs'), distinguishing it from siblings like 'get_job' (individual job) and 'list_jobs' (individual job data). It explicitly contrasts with retrieving individual job data, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Optimized for dashboards and monitoring with minimal network overhead') and when not to ('without retrieving individual job data'), with clear alternatives implied (e.g., 'get_job' or 'list_jobs' for individual data). This helps the agent choose appropriately based on the need for aggregated vs. detailed data.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.