Server Quality Checklist

Tool Scores

• tool_get_eval_summaryC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool returns 'full metadata' and lists components like task info and scoring results, but it doesn't cover critical aspects such as whether this is a read-only operation, error handling, performance characteristics, or data format. For a tool with no annotations, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It front-loads the core purpose in the first sentence, followed by details on returns and parameters. There's no wasted text, though the parameter explanations could be slightly more concise. Overall, it's efficient and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (2 parameters, no annotations, no output schema), the description is moderately complete. It covers the purpose and parameters but lacks output details, error handling, and usage context. Without an output schema, it should ideally explain return values more thoroughly. It's adequate for basic use but has clear gaps for full agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds some value by explaining that 'log_file' is a 'Path to log file (absolute or relative to log_dir)' and 'log_dir' is an 'Optional log directory for relative paths.' However, it doesn't fully document both parameters—missing details like expected file formats, path validation, or examples. With 2 parameters and low schema coverage, this is a minimal but adequate explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get comprehensive evaluation summary including header, results, and stats.' It specifies the verb ('Get') and resource ('evaluation summary') with details about what's included. However, it doesn't explicitly differentiate from sibling tools like tool_get_aggregate_stats or tool_list_logs, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like tool_get_aggregate_stats or tool_search_logs, nor does it specify prerequisites or exclusions. The only implied context is needing an evaluation log file, but this is insufficient for effective tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tool_get_sampleB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes what data is returned but lacks behavioral details: it doesn't specify if this is a read-only operation, potential errors (e.g., invalid paths), performance implications, or how verbose 'include_events' might be. The mention of 'can be verbose' for include_events adds some context, but overall disclosure is minimal for a tool with multiple parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized: it starts with the core purpose, details the return data, and lists parameters with explanations. Every sentence adds value, and it avoids redundancy. Minor improvements could include bullet points for readability, but it's efficient overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, 0% schema coverage, no output schema, and 5 parameters, the description is moderately complete. It covers the purpose and parameters well but lacks behavioral context (e.g., error handling, performance) and doesn't fully address sibling tool differentiation. For a data retrieval tool with multiple inputs, it's adequate but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides a clear 'Args' section explaining all 5 parameters, including defaults and brief semantics (e.g., 'Path to log file', 'Sample ID to retrieve', 'Include event transcript'). This adds significant value beyond the bare schema, though it could elaborate on formats or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get detailed sample data including full conversation history' and specifies what data is returned (input, target, messages, output, scores, metadata). It uses specific verbs ('Get', 'Returns') and identifies the resource ('sample data'). However, it doesn't explicitly differentiate from sibling tools like tool_list_logs or tool_search_logs, which might also retrieve log/sample data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like tool_list_logs or tool_search_logs, nor does it specify prerequisites or contexts for usage. The only implied usage is retrieving detailed sample data, but no exclusions or comparisons are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tool_compare_runsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions what gets compared but doesn't disclose behavioral traits like whether this is a read-only operation, what format the comparison output takes (e.g., table, summary), error handling for invalid log files, or performance considerations. The description is functional but lacks transparency about how the tool behaves beyond its basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It starts with a clear purpose statement, follows with specific comparison details, and ends with parameter explanations. Every sentence adds value with no redundancy or fluff, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (comparing runs with 3 parameters) and lack of annotations/output schema, the description is partially complete. It covers the purpose and parameters well but misses behavioral details (e.g., output format, error handling) and usage guidelines relative to siblings. For a tool without structured metadata, it should do more to compensate, leaving gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides clear semantics for all three parameters: log_file_a and log_file_b as paths to log files, and log_dir as an optional directory for relative paths. This adds meaningful context beyond the schema's basic string types, explaining what each parameter represents and their relationships (e.g., log_dir applies to both files).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Compare metrics between two evaluation runs' with specific details about what gets compared (task/model info, sample counts, score differences, etc.). It uses a specific verb ('compare') and identifies the resource ('evaluation runs'), but doesn't explicitly differentiate from sibling tools like tool_get_aggregate_stats or tool_get_eval_summary that might also involve evaluation data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like tool_get_aggregate_stats or tool_get_eval_summary, nor does it specify prerequisites (e.g., that log files must exist from previous runs). The context is implied (comparing two runs) but lacks explicit when/when-not instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tool_get_aggregate_statsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes what the tool does (get aggregate statistics) and the filtering parameters, but lacks behavioral details such as whether it's read-only, potential performance impacts, error handling, or output format. For a tool with 5 parameters and no annotations, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized, with a clear opening sentence stating the purpose, followed by bullet-like details on statistics and a structured Args section. Every sentence adds value, though it could be slightly more front-loaded by integrating parameter semantics earlier for faster scanning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 parameters, no annotations, no output schema), the description is partially complete. It excels in parameter semantics but lacks behavioral context and usage guidelines. Without an output schema, it should ideally describe return values, but it does not, leaving gaps in understanding the tool's full behavior and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate fully. It provides detailed semantics for all 5 parameters in the Args section, explaining each parameter's purpose (e.g., 'Directory containing log files', 'Filter by task name (supports wildcards)'), including format hints (ISO format for dates) and functionality (wildcard support). This adds substantial value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Get aggregate statistics') and resources ('across multiple evaluation runs'), and it distinguishes from siblings by specifying the type of statistics (summary statistics grouped by task and model). It explicitly lists what statistics are included (success rates, sample counts, token usage totals, duration averages), making it highly specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus the sibling tools (e.g., tool_compare_runs, tool_get_eval_summary). It mentions filtering capabilities but does not explain alternatives or exclusions, leaving the agent to infer usage context without explicit direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tool_list_logsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context: it specifies the file type ('.eval log files'), sorting behavior ('sorted by date, newest first'), and return metadata details. However, it doesn't cover important aspects like error handling, permissions needed to access the directory, or whether the operation is read-only (though implied by 'List').

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It starts with a clear purpose statement, followed by behavioral details, then parameter explanations. Every sentence adds value, though the parameter section is slightly verbose. It's front-loaded with the core functionality, making it efficient for an agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description does a decent job but has gaps. It explains parameters thoroughly and gives behavioral context like sorting and metadata returned. However, it doesn't describe the output format in detail (e.g., structure of returned metadata), error conditions, or dependencies on environment variables, which could hinder agent usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It provides detailed semantics for all three parameters: 'log_dir' (directory with default from env var), 'limit' (max logs with defaults and max value), and 'offset' (pagination). This adds significant value beyond the bare schema, explaining defaults, constraints, and usage context effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List available evaluation log files with metadata.' It specifies the verb ('List'), resource ('evaluation log files'), and scope ('with metadata'). However, it doesn't explicitly differentiate from sibling tools like 'tool_search_logs' or 'tool_get_eval_summary', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'tool_search_logs' (which might filter logs) or 'tool_get_eval_summary' (which might provide summaries), leaving the agent without context for tool selection. Usage is implied only through the action of listing logs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• tool_search_logsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the search/filter functionality and mentions wildcard support, but lacks details on permissions, rate limits, error handling, or what the output looks like (e.g., format, pagination). For a tool with 8 parameters and no output schema, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement followed by a bullet-like 'Args' section. It's appropriately sized for an 8-parameter tool, though the 'Args' formatting could be more integrated. Every sentence adds value, with no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (8 parameters, no annotations, no output schema), the description is partially complete. It excels at parameter documentation but lacks output information, behavioral context, and sibling differentiation. For a search tool with rich filtering, more guidance on result format and usage context would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It provides detailed parameter semantics beyond the schema: it explains each parameter's purpose (e.g., 'Filter by task name'), provides format examples (ISO dates, wildcards), and includes default/limit values for 'limit'. This adds substantial value over the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search and filter evaluation logs by various criteria.' This specifies the verb (search/filter) and resource (evaluation logs). However, it doesn't explicitly differentiate from sibling tools like 'tool_list_logs' or 'tool_get_eval_summary', which likely handle similar log data but with different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'tool_list_logs' or 'tool_get_eval_summary'. It lists filtering capabilities but doesn't explain scenarios where this search tool is preferred over other log-related tools, leaving the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.