Server Quality Checklist

Tool Scores

• run_debateA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description transparently lays out the execution stages, including rounds of critique and the judge's role, which is useful given no annotations. However, it omits side effects, output/return format, and cost/latency implications of invoking multiple providers, and 'one or more rounds' is slightly inconsistent with the schema's minimum of 0.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence covers the full workflow without filler. Every clause contributes meaning about the debate process.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description leaves the return contract ambiguous—whether it returns only the judge's final answer or also the debate transcript. It is otherwise adequate for understanding the tool's purpose and required inputs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75%, and the schema already explains question, rounds, and judge defaults. The description adds process-level context for rounds and the judge but provides no additional parameter-specific detail beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific process—independent answers, critique rounds, and judge synthesis/selection—with a clear verb and resource. It also differentiates the tool from siblings like run_pipeline and run_orchestrator by emphasizing the multi-provider critique workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool: when a question should be answered and cross-validated by multiple AI providers with a final judge. It does not explicitly state exclusions, prerequisites, or when to prefer a sibling pipeline/orchestrator tool instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_pipelineA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the behavioral burden, and it does: it explains the sequential chaining behavior, that each step uses a chosen AI provider, and that step output becomes the next step's input via {{input}}. It does not mention error handling, return format, or provider-specific behavior, but the core execution model is clearly disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence with no filler. It front-loads the primary action ('Runs a sequential pipeline') and then immediately explains the key mechanics, earning every clause without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the pipeline's flow and variable substitution, which are essential. But given the absence of an output schema and annotations, it omits what the tool returns, whether errors occur mid-pipeline, and what happens if a step fails. This is a moderate gap for a multi-step execution tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%, and the description adds meaning by explaining how {{input}} works in promptTemplate and that outputs flow between steps. However, it does not clarify the semantics of the steps array elements (e.g., provider, model, system) beyond what the schema already provides, leaving some gap for low-coverage parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Runs a sequential pipeline' with each step handled by an AI provider and outputs feeding into the next. This distinguishes it from list-type siblings like list_providers and list_models, but it does not explicitly contrast with run_orchestrator or run_debate beyond the sequential chaining concept.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for sequential multi-step AI processing through terms like 'sequential pipeline' and 'feeds into the next step's input.' However, it provides no explicit guidance on when to choose this over sibling tools such as run_orchestrator or run_debate, nor any exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_orchestratorA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden. It openly describes the orchestration flow, the complexity-based specialist cap, and the run report statuses including the evidence label and UNVERIFIED marking. It does not mention potential costs, external calls, or side effects, but the main behavioral contract is clearly disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three dense sentences, each earning its place: the first explains the workflow, the second gives the specialist cap rule, and the third summarizes output behavior. Information is front-loaded and free of filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex multi-agent tool with no output schema, the description covers the execution model, operational constraints, and key output semantics. Minor gaps remain, such as the exact run report structure and provider/model selection details, but those are partially covered by the input schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is high at 80%, so the baseline is 3. The description adds useful orchestration context, such as workers being candidate specialists, synthesizer defaulting to the orchestrator, and the relevance of evidence labels, but it does not map individual parameters in detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs an orchestrator/worker collaboration: a Chief classifies complexity, selects specialists, and a synthesizer merges results. It is specific about the execution model, though it does not explicitly differentiate from sibling tools like run_pipeline or run_debate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool: for complex tasks requiring delegation to multiple specialists and synthesis into a final answer. It does not explicitly state when not to use it or name alternatives, leaving usage guidance implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_providersA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full disclosure burden. It clearly communicates a read-only listing behavior and the exact filtering condition (configured API keys). It does not mention output format or absence of side effects, but for a simple zero-parameter list tool the behavioral disclosure is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one concise sentence with no filler. The action, resource, and qualifying condition are all front-loaded, making it easy for an agent to grasp quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description provides all necessary information: what is listed, which providers are included, and how the result relates to other tools. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the input schema is empty, so the description cannot add parameter-level meaning. The baseline of 4 applies because there are no parameters to misunderstand or document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Lists') with a clear resource (configured AI providers) and names the providers (claude, openai, gemini). It also states the selection criterion (API keys configured) and is clearly distinct from sibling list tools like list_agents and list_models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'usable in the other tools' implies this is a precondition check before running pipeline or orchestration tools, but there is no explicit statement of when to use this tool versus list_agents/list_models or when not to use it. The usage context is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even without annotations, the description discloses an important behavioral trait: it reports codebase-wired capabilities rather than vendor claims. This helps the agent understand the data source and reliability. It does not explicitly state that the tool is read-only or describe side effects, but for a list operation this level of disclosure is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the core purpose, and adds only meaningful distinctions and usage examples. Every sentence earns its place without fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only list tool with no parameters and no output schema, the description gives enough context: what it lists, the source of truth, and two concrete usage cases. It could go further by describing the return format or whether listing is summary-only, but this is not a significant gap for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so parameter semantics are inherently not a burden. The description adds relevant context about what the returned information covers, but no parameter documentation is needed. The baseline score of 4 for zero-parameter tools applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Lists') and resource ('what each model can do'), and adds a distinguishing qualifier: capabilities 'as wired up in this codebase — not what its vendor advertises.' This makes it easy for an agent to understand the tool's unique scope and differentiate it from the sibling list tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear use guidance: 'Use it to see which models support grounded retrieval, and which have verified pricing.' This tells the agent when to invoke the tool, though it does not explicitly mention when not to use it or name alternative tools; the context is sufficient for this simple listing tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_agentsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral burden. It discloses that it is a read/list operation ('Lists'), that provider assignments are current ('currently performs'), and specifies the returned attributes. Minor omissions such as auth or failure behavior are low-relevance for a parameterless query.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core listing semantics and followed immediately by a practical pointer to consumption. No redundant phrasing; even the title is enriched by the 'run_orchestrator' scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 0-parameter list with no output schema, the description covers the returned fields and the integration point (run_orchestrator workers array). Nothing essential is missing for an agent to select and call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters and 100% schema coverage, there is nothing to document; the description correctly focuses on the output and downstream usage. It also links the returned ids to run_orchestrator's workers array, which is more useful than any input-parameter note.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a clear verb ('Lists') and the precise resource ('specialists registered for run_orchestrator'), and enumerates the returned fields. It stands apart from sibling tools like list_providers/list_models by scoping to specialists and their provider assignment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Tells the agent exactly when to call it: before composing run_orchestrator's workers array, using the returned ids instead of redeclaring specialists. It does not state exclusions or when to prefer a sibling, but for a simple registry query the implied context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.