Server Quality Checklist

Tool Scores

• get_provenanceA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the content of the returned provenance record and asserts a completeness guarantee for byte-for-byte reproduction, which is a behavioral claim. However, it doesn't mention operational details like error handling, permissions, or the output format beyond naming RO-Crate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that uses a colon to introduce a list of specific provenance elements. Every word contributes to clarifying the tool's purpose and output, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter, read-only tool, the description sufficiently defines the scope of the returned provenance and its purpose. It gives a detailed list of what the record contains, which is enough for an agent to select it. It lacks details on error scenarios or how the RO-Crate output is delivered, but these are not critical for tool selection given the simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides 100% description coverage for the single run_id parameter, stating it is 'The run id returned by run_pipeline.' The description adds no new meaning about the parameter beyond what the schema already conveys, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool gets 'Workflow Run RO-Crate provenance' for a run, enumerating specific contents (workflow version, container digests, checksums). This specific resource distinguishes it from sibling tools like get_run_status and get_results, which focus on status and outputs rather than full reproducibility metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a full record is needed to reproduce a run byte-for-byte, but it doesn't explicitly compare with alternatives like get_results or state when not to use this tool. The usage context is embedded in the phrase 'needed to reproduce the run byte-for-byte' but lacks explicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_resultsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses the return structure (output files with checksums, size, status) and adds a behavioral note about verifying results independently. This is transparent for a read-only operation, though it doesn't mention potential errors or response format details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the core purpose. Every word adds value—no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description adequately explains returns and usage. It might benefit from a mention of get_run_status to avoid redundancy, but overall it is complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the single parameter run_id is fully documented with its description. The tool description adds no extra parameter semantics beyond what the schema already provides, aligning with the baseline for complete schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: retrieving run outputs including checksums, size, and status. It uses a specific verb ('get') and resource ('outputs of a run'), distinguishing it from siblings like list_pipelines or get_run_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (to get outputs/checksums) but does not explicitly state when to use this tool versus alternatives like get_run_status. No exclusions or alternate tool references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_run_statusA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosure. It clearly indicates a read-only operation (get status) and lists the exact status values. It does not mention error conditions or response format, but for a simple status check, the provided information is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the action and includes all necessary information without any filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter and no output schema, the description effectively covers core behavior, parameter source, and possible return values. It lacks details on error handling or response structure, but these are minor for a status-checking tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully describes the run_id parameter and its origin ('The run id returned by run_pipeline'). The description repeats this origin without adding new semantic detail, so it adds minimal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Get'), a defined resource ('status of a run'), and enumerates the possible status values. It distinguishes itself from siblings by focusing on run status rather than pipeline execution, results, or provenance.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context that run_id comes from run_pipeline, implying usage after starting a run. However, it does not explicitly state when to use this tool versus alternatives like get_results, nor does it mention any exclusions or recommended next steps.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_pipelinesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It discloses the return fields and the read_type enum values, which is useful for a read-only list operation. However, it does not mention potential pagination, ordering, errors, or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action, and every sentence adds value. It is highly concise without unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (no parameters, list operation) and the existence of an output schema, the description is complete. It states the purpose, return contents, and how to proceed, covering all necessary context for correct use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description does not need to explain parameters. The baseline of 4 applies because there is nothing missing in parameter guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and clearly identifies the resource ('available bioinformatics pipelines'). It also specifies what is returned (id, description, read_type) and explicitly distinguishes this discovery tool from describe_pipeline by recommending it as the first step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage guidance: 'Call this first to discover what can be run, then describe_pipeline for details.' This clarifies when to use the tool and gives a clear next step, though it does not enumerate all alternatives or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_pipelineA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description takes on the burden of explaining behavior. It discloses that the run executes asynchronously, returns a run_id, and requires polling for status and results. This is substantial behavioral context, though it could mention failure handling or validity checks.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the primary purpose and follow-up steps. Every clause serves a purpose with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex asynchronous tool with no output schema and no annotations, the description covers initiation, return value, asynchronous behavior, polling, and final result retrieval. It references describe_pipeline for additional detail, making it complete enough for an agent to orchestrate the full pipeline lifecycle.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the schema descriptions already indicate that inputs and params are maps 'per describe_pipeline.' The description adds no meaning beyond the schema, merely restating the same guidance. This meets the baseline but does not elevate it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb and resource: 'Start a pipeline run and return a run_id to track it.' This distinguishes it from sibling tools like list_pipelines, describe_pipeline, and get_run_status, which are explicitly mentioned as complementary follow-ups.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context by instructing to use describe_pipeline for input/param semantics and directing the sequence: poll get_run_status, then fetch get_results and get_provenance. It does not explicitly mention when not to use this tool, but the workflow is well-defined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_pipelineA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden. It discloses the tool's behavior by stating it returns a full manifest containing required inputs, tunable parameters, container image, and output file patterns. However, it does not explicitly state whether the operation is read-only or side-effect-free, though the verb 'return' and the context imply a safe query. The enumeration of output components adds transparency beyond a simple 'describe' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise at two sentences, with the primary function and contents in the first sentence and a usage tip in the second. Every word earns its place, and it is structured with the most important information upfront. No redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description is complete. It specifies the exact components of the returned manifest, compensating for the lack of an output schema. It also provides contextual guidance on when to use it (before run_pipeline). There are no significant gaps given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter pipeline_id is fully described in the schema ('The pipeline id, as returned by list_pipelines'), providing high schema coverage. The tool description does not add parameter-specific details, but since schema coverage is complete, the baseline score of 3 is appropriate. The description adds contextual purpose but not additional semantic meaning for the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Return the full manifest for one pipeline' and enumerates the specific contents. It also distinguishes itself from run_pipeline by positioning itself as a pre-execution step. This makes the purpose unambiguous and distinct from sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Call this before run_pipeline to learn exactly which inputs and params it expects.' This tells the agent exactly when to use this tool and why. The directive is clear and actionable, even if it doesn't explicitly mention when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.