Server Quality Checklist

Tool Scores

• list_runsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns runs with specific fields (iterations, sessions, cost, flagged-call counts, status) and mentions possible statuses. However, it does not explicitly state that the operation is read-only, nor does it describe ordering, pagination behavior, or data freshness. The behavioral context is adequate but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the action ('List loop runs'). It includes necessary detail (explicit vs auto-inferred, example, returned fields) without excessive verbiage. The parenthetical adds context but could be slightly more streamlined. Overall, it is concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one optional parameter and no output schema, the description explains the returned fields (iterations, sessions, cost, flagged-call counts, status) but does not specify the return structure (e.g., array of objects), pagination details (how 'limit' interacts with results, whether there is a next page), or how to interpret 'auto-inferred' runs. It is moderately complete but leaves gaps for an agent to infer.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (the 'limit' parameter is fully described in the schema). The description does not add any additional meaning to the parameter beyond what the schema provides. It does not mention the parameter at all, so the parameter semantics rely entirely on the schema, which is sufficient for a basic optional parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as listing loop runs, specifying the resource ('loop runs') and the action ('List'). It distinguishes from sibling tools like 'list_sessions' by focusing on runs and including run-specific details (iterations, sessions, cost, flagged-call counts, status). The parenthetical notes explicit vs auto-inferred runs, further clarifying scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use the tool (explicit run ID or auto-inferred fresh-context loops) and an example ('Ralph overnight runs'). However, it does not explicitly state when not to use it or mention alternatives among siblings, such as 'get_run_status' for detailed status of a single run. The usage context is clear but lacks exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explainA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Though no annotations exist, the description correctly implies a read-only retrieval operation. It could be improved by explicitly stating idempotency or lack of side effects, but the current text is clear and not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states the action, second lists returned fields. No fluff, front-loaded, and every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 1-parameter tool with no output schema, the description adequately lists return fields (prompt, cost, latency, etc.). Could add format or limit details, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear description for 'action_id'. The tool description adds no additional context to the parameter beyond what the schema already provides, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Retrieve the full captured trace') and the resource ('for a single LLM call'), and enumerates specific returned fields (prompt, system prompt, tool calls, etc.), making it distinct from sibling tools that handle sessions, runs, or search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus siblings like 'get_session' or 'search'. The context that it requires an 'action_id' from a header is implied but not compared to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sessionsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that it returns aggregated stats and lists recent sessions, but does not elaborate on behavioral traits like read-only nature, order, or pagination beyond the limit parameter. This is adequate for a simple list tool but could be more explicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action, and contains no unnecessary words. Every part adds value, earning a top score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one optional parameter and no output schema. The description adequately explains the return content and common use case. It does not detail ordering or filtering beyond 'recent', but given simplicity, it is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with one parameter 'limit' having a clear description. The description adds no extra meaning beyond what the schema already provides, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'List recent agent sessions with aggregated stats' which is a specific verb and resource. It distinguishes from the sibling 'get_session' by noting its use for finding a session_id before calling that tool, thus providing differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to find a session_id before calling get_session or to get a cost overview,' giving clear guidance on when to use. However, it does not discuss when not to use or contrast with other siblings like list_runs or get_run_status.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_run_statusA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It lists the exact data returned (iterations, cost, tokens, flagged calls, completion promise) and implies a read-only operation. No side effects or authentication needs are mentioned, but the simplicity of a status check makes this acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: the first lists the output content concisely, the second provides the use case. Every word adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description adequately explains return values (iterations, cost, tokens, flagged calls, completion promise) and the tool's purpose. It could mention error behavior or format, but as a simple status tool it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%—the only parameter (run_id) is well-described. The description adds no further parameter details beyond identifying the run, so a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'Status of one loop run' with specific data fields (iterations, cost, tokens, flagged calls, completion status). It distinguishes from siblings like list_runs and get_session by focusing on a single run's progress.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states that 'loop runners and agents can use this to decide whether to continue iterating', providing a clear use case. However, it does not mention when not to use it or direct alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses the search scope (all captured calls, specific fields), the return mode (compact summaries with action_id vs full bodies when include_messages is true), and references to 'explain' for deeper inspection. This is useful behavioral context beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first states what the tool does, the second gives usage guidance. It is front-loaded with the core purpose and contains no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of output schema and annotations, the description provides enough to understand what the tool does, what it searches, and what results look like (compact summaries with action_id). It could mention sorting/pagination, but the limit parameter covers one aspect. Overall, it is complete for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of the parameters with descriptions, so the baseline is 3. The tool description itself does not add much parameter-specific meaning—it only reinforces the purpose. The 'include_messages' behavior is described in the schema, not the main description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the verb and resource: 'Full-text search across all captured LLM calls.' It enumerates the searched fields (prompts, outputs, system prompts, agent names, user IDs), which differentiates it from sibling tools that handle sessions or runs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use it: 'Use this to find calls related to a topic, error, or agent.' It also hints at an alternative for drill-in ('to drill in via explain') without naming the tool directly. This gives clear context for selection, though it doesn't explicitly mention when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_sessionA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the default summary format (fields like model, status, tokens, cost), the size implication ('sessions can be megabytes'), and the effect of include_messages. It doesn't explicitly state absence of side effects, but 'Retrieve' strongly implies a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that front-loads the core purpose, then packs valuable caveats and alternatives without fluff. Every phrase earns its place, and the em-dash structure improves readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of an output schema, the description compensates by enumerating summary fields and the condition for full bodies. It also addresses size concerns and directs to explain for individual calls, making the tool's behavior clear enough for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value by explaining why include_messages defaults to false (session size) and what the summary contains, which complements the schema's boolean description. It reinforces the session_id's source but doesn't introduce conflicting semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb-resource pair: 'Retrieve the LLM calls of an agent session' and adds 'in chronological order,' which clearly scopes the operation. It distinguishes from siblings by contrasting with the explain tool for single calls and implying list_sessions for session listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use a different tool: 'single calls are better fetched via the explain tool using the action_id from a summary row.' It also provides usage context for the include_messages parameter, warning about large payloads and explaining the default compact summary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.