Server Quality Checklist

Tool Scores

• get_jobA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds valuable context beyond this: it specifies that the output includes 'all options displayed in a table showing required status, defaults, and allowed values', which clarifies the return format and depth of information provided. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, returns, examples) and front-loaded key information. It's appropriately sized, but the example section could be slightly condensed as it repeats the purpose. Most sentences earn their place by adding value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter), rich annotations, and presence of an output schema, the description is largely complete. It explains the parameter, output format, and includes an example. However, it doesn't mention error cases or prerequisites, leaving minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by explaining the single parameter 'job_id' as 'The job UUID', adding semantic meaning beyond the schema's title 'Job Id'. This clarifies the expected format and purpose, though it doesn't detail validation or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get detailed information about a specific job' with the verb 'get' and resource 'job'. It distinguishes from siblings like 'list_jobs' (which lists multiple jobs) by specifying retrieval of a single job, but doesn't explicitly contrast with 'get_execution' or 'get_execution_output'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when detailed information about a specific job is needed, as opposed to listing jobs. However, it doesn't explicitly state when to use this versus alternatives like 'get_execution' or provide exclusions (e.g., not for modifying jobs). The context is clear but lacks explicit guidance on sibling differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_executionA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds value by specifying the return content ('full execution details including status, timing, node results, and the arguments used'), which provides useful context beyond annotations. No contradiction with annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, returns, args, examples), front-loaded with the core purpose, and every sentence adds value without redundancy. It efficiently conveys necessary information in a compact format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter), rich annotations (covering safety and idempotency), and the presence of an output schema (implied by 'Returns: Execution object'), the description is complete enough. It explains the purpose, parameter, return content, and includes examples, addressing all key aspects without overloading.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description includes an 'Args' section that documents the single parameter (execution_id) as 'The execution ID (integer)', adding meaning beyond the schema's basic type. However, it does not elaborate on format constraints or examples beyond the integer type, so it partially compensates for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('detailed information about a specific execution'), distinguishing it from siblings like list_executions (which lists multiple executions) and get_execution_output (which focuses on output rather than full details). The purpose is specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying it retrieves details for 'a specific execution' (via execution_id), but it does not explicitly state when to use this tool versus alternatives like list_executions or get_execution_output. No exclusions or prerequisites are mentioned, leaving some ambiguity for the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_executionsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  While annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, the description adds valuable behavioral context beyond annotations: it specifies that results are ordered by start time (most recent first), mentions filtering capabilities, and provides concrete examples of usage patterns. This enhances the agent's understanding of how the tool behaves in practice.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by clear sections for Args, Returns, and Examples. Every sentence adds value: the opening establishes purpose, the second explains filtering and ordering, and the examples demonstrate practical usage. No wasted words or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that annotations cover safety aspects (read-only, idempotent, non-destructive), an output schema exists (implied by 'Returns: List of Execution objects'), and the description provides good usage guidance with examples, this description is complete enough for the agent to understand and correctly invoke this tool. The combination of structured data and descriptive text covers all essential aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage (the single 'query' parameter lacks a description in the schema), but the description compensates by explaining that filtering can be done by 'project, job_id, status, or time range' and provides examples showing how to construct ExecutionQuery objects. However, it doesn't document all available parameters (like user, older_filter, limit, offset) that appear in the schema's nested object.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List job executions') and resource ('executions'), distinguishing it from sibling tools like get_execution (which retrieves a single execution) or list_jobs (which lists jobs rather than executions). The opening sentence establishes the core purpose with optional filtering capabilities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('List job executions with optional filtering') and includes examples showing different filtering scenarios. However, it doesn't explicitly state when NOT to use it or mention alternatives like get_execution for single execution retrieval, though the sibling tool names provide some implicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_jobsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable behavioral context beyond what annotations provide. While annotations indicate read-only, idempotent, and non-destructive operations, the description reveals that results are returned as a 'numbered markdown table' and explains how to reference jobs using the '# column' in subsequent commands. This output format guidance is crucial for agent interaction but doesn't cover aspects like rate limits or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise. It begins with a clear purpose statement, immediately explains the return format and how to use it, then provides parameter documentation and practical examples. Every sentence serves a specific purpose with zero wasted words, and the information is front-loaded appropriately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that this is a read-only listing tool with comprehensive annotations (readOnlyHint, idempotentHint, destructiveHint), a well-documented input schema, and an output schema indicated in context signals, the description provides exactly what's needed. It explains the purpose, output format, parameter usage with examples, and how to interpret results for subsequent operations - making it complete for this tool's complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions 'optional filtering' and documents the single 'query' parameter with examples showing different filtering scenarios. However, with 0% schema description coverage in the context signals, the detailed parameter documentation (project, group_path, job_filter, etc.) comes entirely from the input schema. The description provides basic guidance but doesn't add significant semantic value beyond what's already in the well-documented schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List jobs in a Rundeck project with optional filtering.' It specifies the verb ('List'), resource ('jobs'), and context ('in a Rundeck project'), distinguishing it from sibling tools like get_job (which retrieves a single job) or list_executions (which lists executions rather than jobs).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: for listing jobs with optional filtering in a Rundeck project. It includes examples that demonstrate different filtering scenarios. However, it doesn't explicitly state when NOT to use it or mention alternatives like get_job for single job retrieval, which would have earned a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_execution_outputA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds valuable behavioral context beyond annotations: it explains how to handle running executions with polling using offset, mentions the 'completed' field to indicate execution status, and describes the return structure (ExecutionOutput with log entries and metadata). No contradictions with annotations exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized: it starts with a clear purpose statement, provides usage guidelines, lists parameters with explanations, describes returns, and includes practical examples. Every sentence adds value, with no wasted words, and the information is front-loaded for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, polling behavior) and the presence of an output schema (which handles return values), the description is complete. It covers purpose, usage, parameters, behavioral traits, and provides examples, making it sufficient for an agent to select and invoke the tool correctly without needing additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage (titles only, no descriptions), the description carries the full burden of explaining parameters. It effectively adds meaning: it explains 'execution_id' as 'The execution ID (integer)', 'last_lines' as 'Return only the last N lines (overrides offset)', 'max_lines' as 'Maximum number of lines to return from offset', 'offset' as 'Byte offset to start reading from (for tailing)', and 'node' as 'Filter output to a specific node'. This compensates fully for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Get the log output from a job execution' and 'Retrieves log entries from the execution', which is a specific verb+resource combination. It distinguishes itself from sibling tools like 'get_execution' (which likely returns execution metadata rather than logs) and 'list_executions' (which lists multiple executions).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'For running executions, use the 'offset' parameter to poll for new output.' It also offers practical examples showing when to use different parameters (e.g., getting all output vs. last lines vs. tailing), which implicitly guides the agent on when to use this tool versus alternatives like checking execution status through other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.