Server Quality Checklist

Tool Scores

• explain_resultA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well by explicitly stating 'Read-only'. It also discloses what the explanation covers (gates, failed cases, per-assertion and claim-level breakdown), giving a clear picture of the tool's behavior and output without revealing internals.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two tight sentences, front-loaded with the primary action ('Read... and explain why it passed or failed') and structured to list the explanation contents. The final 'Read-only' is a concise, valuable note with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema or annotations, the description sufficiently explains what the tool does and what it returns. It covers read-only behavior and the scope of the explanation, which is adequate for a simple read tool. It doesn't discuss error cases or relationship to siblings, but that's not essential.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions cover 100% of parameters, so baseline is 3. The description adds the artifact path example (.evalgate/result.json) but does not go beyond the schema's per-parameter descriptions for resultPath or failuresOnly. It provides no additional meaning for the failuresOnly parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's verb and resource: 'Read an evalgate result artifact' and 'explain why it passed or failed'. It also details specific output elements (which gates tripped, cases failed, per-assertion explanations, claim-level grounding breakdown), fully distinguishing it from siblings like list_suites or run_suite.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after a run (explaining an existing result artifact) but does not explicitly state when to use it versus alternatives or mention any exclusions. It lacks a direct 'use this instead of X' or 'after run_suite' guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_driftA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry behavioral disclosure. It clearly states 'Read-only' and describes the output (per-series delta and slope), which covers the key behavioral traits. It does not document file-missing or empty-history behavior, but for a read-only analysis tool this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four short sentences, each earning its place: the resource and purpose, the rationale, the output, and the read-only nature. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema, it covers purpose, output, and safety. It lacks explicit handling of edge cases (e.g., missing file, no drift), but the available schema and description provide sufficient context for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage with descriptive comments for all three parameters, so baseline is 3. The description does not add any parameter-specific details beyond noting the history file path and the concept of slow decline, which is already captured by the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Analyze') and resource ('.evalgate/history.jsonl') with a clear purpose: detecting slow decline that per-PR gates miss. This differentiates it from siblings like run_suite or judge_status, which focus on individual results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for detecting cumulative performance drift over time rather than per-PR checks, noting that individual gates can miss gradual declines. Does not explicitly name alternatives or exclusions, but the contrast with 'per-PR regression gate' provides clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• judge_statusA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It states 'Read-only,' which is a critical safety trait, and explains that it reads a specific file and reports agreement. It does not mention failure modes or return format, but for a simple status read, this is sufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the core action, and uses the metaphor 'random number generator with good manners' to add context without wasted words. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple status-check tool with full schema coverage and no output schema, the description provides sufficient context: it names the file, explains what the report contains, and gives usage timing. It could describe the output shape in more detail, but the core purpose is fully covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both calibrationPath and judgeId described in the input schema. The description adds no additional parameter-level semantics, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Read .evalgate/calibration.json and report the judge’s measured agreement with human scores.' This distinguishes it from sibling tools like list_suites, explain_result, check_drift, and run_suite, which all focus on different aspects of the tool ecosystem.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage guidance: 'call this before trusting any rubric or grounded score.' This tells the agent when to use the tool, though it does not mention alternatives or when not to use it, which would earn a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_suitesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of safety disclosure. It explicitly states 'Read-only — no model calls, no cost', which is important behavioral info. It also mentions 'validate', implying it may error on invalid input, but doesn't detail error behavior. Overall, good transparency for a simple tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: first states the action and return values, second provides a safety note. Every word adds value, and the key information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one simple parameter, no output schema, and no annotations. The description compensates well by listing the return components and explicitly stating the read-only behavior. Nothing critical is missing for an agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter 'suitePath', and the description only restates the same meaning ('directory or file') without adding extra details like formats or examples. Baseline 3 is appropriate since the schema already documents the parameter fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it loads and validates eval suites, and enumerates exactly what it returns (suite name, cases, assertions, threshold policy). This distinguishes it from siblings like run_suite, which would execute suites.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The read-only note ('no model calls, no cost') provides clear context for when to use this tool—for inspection rather than execution. It does not explicitly name alternative tools, but the context is clear enough for an agent to select it appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_suiteA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and explicitly warns 'EXECUTES the SUT module and may call model providers, so it can incur real cost' and states 'Writes nothing — the result is returned in memory, not saved as an artifact.' This discloses side effects and persistence behavior, which is crucial for an agent deciding to run it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first states the core function, the second delivers essential warnings about cost and non-persistence. Every sentence carries meaningful information with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the four parameters are fully documented in the schema and the description covers side effects and return notion, the tool is largely complete. However, the absence of an output schema makes the vague 'scored result' less informative than it could be, and there is no guidance on how to interpret the result or error cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All four parameters have detailed descriptions in the schema (100% coverage), so the baseline is 3. The description itself only reiterates 'suites at a path' and 'system-under-test module' without adding new semantics for concurrency or calibrationPath, so it adds no significant value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it 'Run the suites at a path against a system-under-test module and return the scored result,' identifying the specific action (run), the resource (suites at a path + SUT module), and the output (scored result). This distinguishes it from siblings like list_suites and explain_result.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for what the tool does, and the SUT execution and cost note implies when to use it (when you need to actually evaluate a module). However, it does not explicitly compare with alternatives or state when not to use it, leaving the agent to infer from sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.