Server Quality Checklist

Tool Scores

• get_workflow_healthA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes what the tool returns but fails to state whether it is read-only, destructive, or any side effects. No mention of auth requirements, rate limits, or data freshness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first lists contents, the second provides use cases. No redundant or extraneous information. Efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description partially compensates by listing metric categories, but it does not describe the structure or format of the report (e.g., JSON fields). Adequate but could be more explicit about the output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description lists metrics but does not add parameter-specific details beyond what the schema provides. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description precisely states the tool returns a combined health report for a specific workflow, listing concrete metrics (duration, success, MTTR, slowest/failed tests, flaky tests). It clearly distinguishes from siblings (get_bottlenecks, get_config, etc.) which focus on other aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises when to use the tool: 'Use this to evaluate test splitting opportunities, flakiness, and overall reliability.' It provides clear context but does not mention when not to use it or alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_bottlenecksA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description discloses key behaviors: it fetches Insights data per workflow and per job, and sorts by total_credits_used. This adds valuable context beyond a mere operation, though it lacks mentions of rate limits or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loading the core purpose in the first sentence and providing usage context in the second, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While lacking an output schema, the description explains the sorted result based on total_credits_used, which is sufficient for an agent to understand the tool's return. It could specify output structure but remains adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents each parameter. The description adds high-level purpose but does not elaborate on parameter semantics beyond what the schema provides, meeting the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb-resource pair ('Identify the slowest and most credit-intensive jobs') and clearly distinguishes this tool from siblings like get_config, get_dag, etc., which cover other aspects of a project.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states concrete use cases ('find resource_class over-provisioning, unnecessary serialisation, and credit hot-spots'), providing clear guidance on when to employ the tool, though it omits explicit when-not-to-use scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_configA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It describes a non-destructive fetch operation (reads config, returns YAML) and implies no side effects. It does not mention authentication or rate limits, but the read-only nature is clear from 'Fetch' and the return description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each concise and impactful. The first sentence states the action and scope, the second explains the return format, and the third gives a usage tip. No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description adequately explains the return format (source and compiled YAML). It covers the main use case and provides a starting point for review. Minor gaps like error handling or edge cases are not critical for this fetch tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the baseline is 3. The description adds no new meaning beyond what the schema already provides for branch, pipeline_id, and project_slug; it merely restates the schema's logic.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fetch the CircleCI config for a project's latest pipeline (or a specific pipeline)' with specific verb and resource. It distinguishes from siblings like get_dag or validate_config by positioning itself as the starting point for config review.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes the directive 'Use this as the starting point for any config review,' providing clear context on when to use the tool. However, it does not explicitly exclude alternative tools or mention cases where siblings would be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dagA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It mentions the optional include_resource_class parameter affects performance (N API calls), but does not disclose other behavioral traits like authentication needs or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose, second gives usage guidance. No wasted words; each sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 parameters and no output schema, the description covers key behavioral aspects. It could mention the return format (e.g., graph structure), but current level is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description adds meaning by explaining what the tool returns when include_resource_class is true (actual resource_class). This goes beyond the schema's parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns the job dependency graph (DAG) for a pipeline or workflow, which distinguishes it from sibling tools like get_bottlenecks or get_config.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides specific use cases: 'identify unnecessary serialization, missing requires, and executor sizing.' It does not explicitly list when not to use or alternative tools, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_configA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the return of validity, errors, and fully-expanded YAML (with orbs resolved), which is comprehensive. It does not mention that the tool has no side effects, but that is implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose. Each sentence adds value without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema or annotations, the description explains return values (validity, errors, expanded YAML) and the purpose clearly. It could mention output format but is still fairly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds minimal context beyond schema, e.g., owner_id resolves private orbs. Overall, the schema already documents parameters adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates and compiles a CircleCI config YAML string, specifying the verb (validate/compile), resource (config YAML), and output (validity, errors, expanded YAML). It distinguishes from sibling tools like get_config which likely retrieves config.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly recommends using this tool to verify a proposed config change before recommending it, providing clear context. It does not explicitly list when not to use or alternatives, but the recommendation is specific enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.