Server Quality Checklist

Tool Scores

• relay_runs_listC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool is for listing runs 'for debugging and reference', which implies read-only behavior, but doesn't specify details like pagination, sorting, error handling, or authentication requirements. This leaves significant gaps for a tool that likely interacts with workflow data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that states the core purpose upfront. It avoids unnecessary words, though it could be slightly more structured by separating purpose from usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what the output looks like (e.g., list format, fields included), how 'recent' is defined, or other behavioral aspects needed for effective use. For a tool with potential complexity in workflow runs, this is insufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'limit' parameter well-documented. The description doesn't add any parameter-specific information beyond what the schema provides, so it meets the baseline score of 3 for adequate but no extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('recent workflow runs'), making the purpose understandable. However, it doesn't explicitly differentiate from sibling tools like 'relay_run_get' or 'relay_workflow_run', which likely have related but distinct purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance with 'for debugging and reference', but it doesn't specify when to use this tool versus alternatives like 'relay_run_get' or 'relay_workflow_run'. No explicit when-not-to-use or prerequisite information is included.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relay_run_getB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions that the tool retrieves 'full details' including 'step outputs and trace URL', which adds some context about what information is returned. However, it doesn't describe other behavioral traits, such as whether this is a read-only operation (implied by 'Get' but not stated), error handling, rate limits, or authentication needs. For a tool with zero annotation coverage, this leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Get full details of a specific run') and adds specific inclusions ('including all step outputs and trace URL'). There is no wasted text, and every word earns its place by clarifying the tool's scope and output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (retrieving detailed run data) and lack of annotations and output schema, the description is partially complete. It specifies what details are included (step outputs, trace URL), which helps, but doesn't cover other aspects like return format, error cases, or how it differs from siblings. Without an output schema, more detail on the response would be beneficial, but the description provides a basic understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the single parameter 'runId' documented as 'The run ID to retrieve'. The description doesn't add any meaning beyond this, as it doesn't explain where to obtain the run ID or its format. With high schema coverage, the baseline is 3, and the description doesn't compensate with extra param details, so it meets the minimum viable level.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get full details of a specific run including all step outputs and trace URL.' It specifies the verb ('Get'), resource ('run'), and scope ('full details'), distinguishing it from siblings like 'relay_runs_list' (which likely lists runs) and 'relay_run' (which might be more basic). However, it doesn't explicitly differentiate from 'relay_workflow_run', which could be a similar tool for workflows.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing a run ID from another tool, or compare it to siblings like 'relay_run' or 'relay_workflow_run'. The context is implied (use when you have a run ID and want detailed info), but no explicit usage rules or exclusions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relay_models_listA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses key behavioral traits: the tool returns pricing information and clarifies RelayPlane's BYOK (Bring Your Own Key) model with no API usage charges. However, it doesn't mention response format, pagination, or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste: the first states purpose and key attributes, the second clarifies pricing context. It's front-loaded with essential information and appropriately sized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one optional parameter and no output schema, the description is reasonably complete. It covers purpose, usage context, and pricing model, though it could benefit from mentioning response structure or example output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the optional 'provider' parameter with its enum values. The description doesn't add any parameter-specific details beyond what the schema provides, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('available AI models') with specific attributes ('capabilities and pricing'). It distinguishes from siblings by focusing on model metadata rather than execution or workflow tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit context for when to use ('to check valid model IDs before testing'), which helps guide selection. However, it doesn't mention when NOT to use this tool or name specific alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relay_skills_listA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that this is a read operation (list/returns) and describes the return format (skill names, descriptions, metrics, examples), which is helpful. However, it doesn't mention potential limitations like pagination, rate limits, authentication requirements, or whether the list is comprehensive versus filtered by default.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences that efficiently convey purpose, resource definition, and return details without redundancy. Each sentence adds value: the first defines the action and resource, the second specifies the return format. It's front-loaded and wastes no words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one optional parameter, no output schema, no annotations), the description is reasonably complete. It explains what skills are and what data is returned, which compensates for the lack of output schema. However, for a list operation, it could benefit from mentioning potential behavioral aspects like ordering or default behavior when no category is specified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents the single optional parameter 'category' with its enum values. The description doesn't add any parameter-specific information beyond what's in the schema, but with only one parameter and high schema coverage, the baseline is 3. The description's explanation of skills provides context that indirectly supports understanding the category filter, warranting a slight bump to 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('available pre-built workflow skills'), defines what skills are ('reusable patterns for common tasks'), and distinguishes this from siblings like relay_models_list or relay_runs_list by focusing specifically on skills rather than models or runs. It provides specific examples of skill types (invoice processing, content pipelines).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by explaining what skills are and what information is returned, but it doesn't explicitly state when to use this tool versus alternatives like relay_models_list or relay_workflow_validate. There's no guidance on prerequisites or when-not-to-use scenarios, leaving usage decisions to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relay_workflow_runA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: intermediate results stay in the workflow engine (not the agent's context), provides 90%+ context reduction, and clarifies cost tracking (provider bills, not RelayPlane fees). However, it doesn't mention error handling, timeouts, or output format details, leaving some behavioral aspects unclear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the core purpose in the first sentence. All sentences add value: context reduction benefits, usage guidelines, and cost clarification. However, the cost explanation could be slightly more concise, and the structure might benefit from clearer separation of key points.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (multi-step workflows with nested objects) and no annotations or output schema, the description does well to cover purpose, usage, and key behavioral traits. It addresses context reduction and cost tracking, which are critical for this type of tool. However, without an output schema, it doesn't describe return values or error formats, leaving some gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal parameter semantics beyond the schema—it mentions 'multi-step AI workflow' which aligns with the steps parameter, and references input accessibility via {{input.field}}, but doesn't provide additional syntax or usage details. Baseline 3 is appropriate when the schema does most of the work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Execute a multi-step AI workflow' with specific details about intermediate results staying in the workflow engine and providing 90%+ context reduction. It distinguishes from siblings by emphasizing multi-step workflows requiring multiple model calls or tool integrations, unlike simpler run or list tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Use for any task requiring multiple model calls or tool integrations.' This clearly indicates when to use this tool versus simpler alternatives like relay_run (likely for single-step execution) or list tools. It also clarifies cost implications, helping users decide based on billing considerations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relay_runA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it returns output, token usage, estimated provider cost, and trace URL, and clarifies cost tracking (provider bills, not RelayPlane fees). However, it doesn't mention rate limits, authentication needs, or error handling, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by usage context and important behavioral notes. Every sentence adds value: the first states the action, the second provides usage guidance, and the third clarifies cost details. It's concise with zero wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description does well to cover purpose, usage, and key behavioral traits like return values and cost tracking. However, it lacks details on error cases, response format beyond listed items, or performance characteristics, which could be useful for a tool with 4 parameters and no structured output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema, such as examples or constraints. Baseline 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Execute a single AI model call') and resource ('AI model'), distinguishing it from sibling tools like relay_models_list (list models) and relay_workflow_run (execute full workflows). It explicitly mentions testing prompts before building workflows, which helps differentiate its use case.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Useful for testing prompts before building full workflows'), implying alternatives like relay_workflow_run for production workflows. It also notes cost tracking specifics, helping users understand appropriate contexts for usage versus other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• relay_workflow_validateA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: it's a validation-only operation ('validate workflow structure'), has no cost ('free'), avoids LLM calls, and specifies what checks are performed and omitted. However, it doesn't mention error handling, response format, or performance characteristics, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by specific checks and exclusions, all in two efficient sentences with zero wasted words. Each sentence earns its place by clarifying scope and usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (validation with specific checks), no annotations, and no output schema, the description is largely complete for guiding usage. It covers purpose, limitations, and alternatives well. However, it lacks details on output format or error responses, which could be useful for a validation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter 'steps' well-documented in the schema. The description adds minimal value beyond the schema by referencing 'same format as relay_workflow_run', which provides context but no additional semantic details. This meets the baseline of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('validate workflow structure') and resource ('workflow'), distinguishing it from siblings by explicitly contrasting with 'relay_workflow_run' for full validation. It specifies what is checked (DAG structure, dependency references, model ID format) and what is not (schema compatibility, prompt effectiveness), providing precise scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('validate workflow structure without making any LLM calls (free)') and when not to ('Does NOT validate schema compatibility between steps or prompt effectiveness'), with a clear alternative named ('use relay_workflow_run for full validation'). This provides complete guidance on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.