Server Quality Checklist

Tool Scores

• fleet_session_roundA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the core behavior (submits scores, evaluates, returns snapshot) and response structure, making it minimally adequate. However, without annotations, it does not mention potential side effects, error conditions, or auth requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences) and well-structured: first sentence states action and mechanism, second defines response shape and downstream connections. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 required params, no output schema), the description provides sufficient context for use, including the response structure and integration with sibling tools. Minor gaps like error handling are acceptable for this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are already documented. The description adds context about cumulative evaluation, but does not enrich parameter semantics beyond the schema. Baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('submit a scoring round'), the resource ('existing fleet session'), and distinguishes from sibling tools like fleet_session_create or fleet_evaluate by specifying that it submits a round to an existing session.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives. It only mentions downstream usage (episodes can be passed to visualize, alert_check, etc.), but lacks conditions or exclusions for selecting this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_configA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries burden. It discloses that no API call, auth, or credits are involved, and that the instruction field guides code generation. However, it omits details like idempotency or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no wasted words. Every sentence contributes value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description details response structure (fields). With only 2 parameters, coverage is good. Missing usage guidelines but otherwise complete for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline 3. Description adds 'no API call, no auth, no credits' which is not parameter-specific. Parameter descriptions in schema already cover framework examples and use_case enum values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it generates CI-1T integration boilerplate, specifies no API call, no auth, no credits, and outlines the response structure. It is distinct from siblings like evaluate or health.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for generating boilerplate but lacks explicit when-to-use or alternatives. No exclusions or comparisons to other tools are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_api_keysA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It describes the operation as listing keys and gives response format, but does not disclose behavioral traits like authentication requirements, rate limits, or behavior when no keys exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two brief sentences: the first states purpose, the second provides response format and a usage hint. No extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool with one required parameter and no output schema, the description includes the response structure. However, it does not address edge cases like empty results or pagination, but remains sufficient for basic use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter user_id, which is already described in the schema. The description adds no additional meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all API keys for the authenticated user', specifying the verb, resource, and scope. It also includes response format, distinguishing itself from sibling tools like create_api_key and delete_api_key.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a usage hint directing to use record id with delete_api_key for revocation. However, it does not explicitly state when to use this tool vs alternatives or exclude inappropriate contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fleet_session_listA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the burden. It discloses that only active sessions are listed and gives the response structure, but does not mention side effects, authentication needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence with an inline response example, front-loading the core purpose without any wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, no output schema, and low complexity, the description is complete: it states what it does (list active sessions) and what the response looks like, which is sufficient for an end-user to use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and schema coverage is 100% trivially. Per the rules, a baseline of 4 applies because the description has no param info to add, but its absence is not a flaw.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'all active fleet sessions', and distinguishes from siblings like fleet_session_create or fleet_session_delete by specifying it is a listing operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives such as fleet_session_state or other listing tools; no conditions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_api_keyA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior. It states that the API key cannot be retrieved after creation and specifies the response structure, which are important behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loads the purpose, and includes a critical warning. No redundant or unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains the response structure and includes the essential 'save the key' warning. It covers the creation aspect well, though could mention authentication or rate limits if applicable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents all parameters. The description does not add extra meaning beyond what the schema provides, hence a baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create a new CI-1T API key') and lists the response fields. It distinguishes itself from sibling tools like delete_api_key and list_api_keys by focusing on creation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description lacks explicit guidance on when to use this tool versus alternatives such as list_api_keys or delete_api_key. It provides a critical note about saving the key but no usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fleet_session_stateA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry behavioral disclosure. It explains the read-only nature and response structure, but omits details like error handling or prerequisites.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise, front-loaded sentences with no superfluous information, efficiently conveying purpose and usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequately covers purpose, usage, and response format for a simple tool, though it could be improved by mentioning potential error states or prerequisites.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, the description adds no additional semantic value beyond what the schema already offers for the single parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('get') and resource ('current state of a fleet session'), and distinguishes it from sibling tools like fleet_session_create or fleet_session_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises using it to inspect accumulated results between rounds, providing clear context but lacking explicit when-not-to-use or alternative pointers.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fleet_session_deleteA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the burden. It discloses the response shape and that the tool frees server resources, indicating a destructive operation. Missing details on error handling or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no filler. The essential information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple deletion tool with one parameter and no output schema, the description covers purpose, usage, and response. Could mention error cases but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the parameter 'session_id' is described in the schema. The description does not add new parameter semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Delete'), the resource ('a fleet session'), and the identifier ('by ID'). It distinguishes from sibling tools like fleet_session_create and fleet_session_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises when to call ('Call when monitoring is complete to free server resources'), providing context. However, it does not specify when not to use or describe alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_windowsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that no API call, auth, or credits are required—key behavioral traits. It also details the response format. However, it omits potential constraints like data size limits or performance characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with no wasted words. It front-loads the core purpose, then provides key restrictions, data source, and a structured response example. Every sentence serves a clear function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description compensates with a detailed response structure. It explains when to use the tool, what inputs are expected, and what outputs to anticipate. For a comparison tool, this is thorough and sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the parameter descriptions add some context (e.g., baseline episodes from a known-good run, recent episode array). The broader description clarifies that these arrays come from evaluate responses. This adds marginal value beyond the schema but does not provide detailed formatting or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: comparing two windows of episodes to detect drift or degradation. It uses a specific verb ('Compare'), specifies the resource ('windows of episodes'), and highlights that it requires no API call, no auth, no credits. This distinguishes it from sibling tools like 'evaluate' or 'fleet_evaluate'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly recommends using this tool after multiple evaluate calls to track model health over time, and notes that inputs come from evaluate or fleet_session_round responses. While it provides clear context for use, it does not explicitly state when not to use it or mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_api_keyA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavior. It states deletion and response format but does not mention irreversibility, authorization needs, or side effects. Adequate for a simple delete operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences covering purpose, usage hint, and response format. No redundancy or unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter deletion tool, the description provides all essential information: what it does, what input is needed, and what response to expect. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already describes the 'id' parameter well, and the tool description adds value by specifying the source (from list_api_keys), aiding correct invocation. Schema coverage is 100%, so baseline 3 incremented to 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb+resource (delete API key) and specifies how to identify it (by PocketBase record ID from list_api_keys), distinguishing it from sibling tools like create_api_key and list_api_keys.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides a prerequisite: ID must come from list_api_keys. No explicit when-not-to-use or alternatives, but context implies using after listing keys.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_invoicesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description bears full burden. It discloses the response structure (invoices array with fields, has_more, cursor) and pagination mechanism, which is sufficient transparency for a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two brief sentences, no wasted words, front-loaded with purpose followed by response format and usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple paginated list tool with one optional parameter and no output schema, the description provides a clear response structure and usage guidance, fully covering the needed context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with the cursor parameter already described. The description reinforces its role in pagination ('Pass cursor to paginate') but adds no new semantic details beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get billing history (Stripe invoices)' with a specific verb and resource. It distinguishes from sibling tools like 'alert_check' or 'convert_scores' by focusing on invoices.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly mentions pagination usage: 'Pass cursor to paginate'. It does not explicitly state when to avoid or alternative tools, but the purpose is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• visualizeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full transparency burden. It clearly states 'no API call, no auth, no credits', indicating a safe, read-only operation. It also describes the output structure. It lacks details on error handling or limitations, but overall provides sufficient behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each serving a distinct purpose: purpose and key constraints, input, output, and usage instructions. No redundant or extraneous text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters, no output schema, and no annotations, the description covers all necessary context: input source, output format and contents, and post-use action. It is fully sufficient for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds context that 'episodes' comes from an evaluate or fleet_evaluate response and the default title, which is helpful but not substantial beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates an interactive HTML visualization of CI-1T evaluate results, distinguishing itself from sibling tools like evaluate or fleet_evaluate. It specifies the input (episode array) and output (file path with detailed contents), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says it takes an episode array from evaluate or fleet responses, guiding the agent on when to use it (after evaluation). It also instructs to open the file in a browser or VS Code. However, it doesn't explicitly state when not to use or mention alternatives, which prevents a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• convert_scoresA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description discloses no API call, no auth, no credits, and shows the response structure. It lacks details on error handling for out-of-range inputs, but for a simple conversion this is mostly sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single clear paragraph with front-loaded purpose, response format, and usage hints. Every sentence adds value with no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool with no output schema, the description covers conversion logic, response structure, and usage. It omits boundary error behavior, but overall it is adequately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description adds context by specifying the numeric ranges and response fields. It reinforces the meaning of the parameters without repeating the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool converts between probability floats and Q0.16 integers, with exact ranges and response format. It distinguishes itself from sibling tools by noting it's a no-API, no-auth, no-credit utility.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage hints: 'Use to_q16 before evaluate, from_q16 to make CI outputs human-readable.' It implies the tool is a quick conversion step, but does not elaborate on when to avoid it (e.g., if inputs are already in the correct format).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• evaluateA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, description details the engine call, auto-conversion, response structure, and classification, providing solid behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Dense but well-organized single paragraph; front-loads purpose and provides essential details without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description fully explains response structure and classification, plus chaining guidance, making it complete for evaluation context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds value beyond schema by explaining auto-conversion for scores and default value for n; high schema coverage is compensated.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb+resource: 'Evaluate prediction stability.' Specifies input, output, and chaining to sibling tools, distinguishing its role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explains input types, output structure, classification thresholds, and suggests chaining to specific sibling tools for further analysis.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• probeA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully bears the burden and does an excellent job: it explains that the tool sends the same prompt 3 times, uses similarity methods, and describes two modes with different credit and authentication needs. It also details the response format and how to chain with evaluate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections, each sentence adds information. It is slightly lengthy but appropriately so for the complexity. It front-loads the purpose and then explains modes and response.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters and no output schema, the description is quite complete: it explains modes, response format, and how to use with evaluate. It could mention potential errors or limitations, but overall it provides sufficient context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage, but the description adds significant value: it explains the two modes (BYOM vs Default), how base_url triggers BYOM, that model_api_key is sent as Bearer token, and the purpose of each parameter beyond the schema's basic descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Probe an LLM for prediction instability. Sends the same prompt 3 times and compares responses...' It uses specific verbs and resources, and distinguishes itself from siblings like evaluate and convert_scores.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides two modes (Default and BYOM) with clear conditions for each, and mentions that returned scores can be passed to evaluate. While it doesn't explicitly state when not to use, the context is sufficient for an agent to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• alert_checkA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations, but description fully covers behavior: no external calls, details response shape (status, alerts list, thresholds), lists alert types. Very transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise yet packed with useful info: purpose, input source, response structure, alert types. No fluff, well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a non-API-checking tool with 6 params and no output schema, description covers inputs, outputs, alert types, thresholds, and constraints. Fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  100% schema coverage so baseline 3. Description adds default thresholds and context (e.g., '0.45 = Drift boundary') beyond property descriptions, and explains overall purpose that ties parameters to alert types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Check episodes against configurable thresholds and return triggered alerts' – specific verb and resource. It also distinguishes from siblings like evaluate/fleet by stating input source.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly notes no API call/auth/credits, and input from evaluate or fleet. Suggests lightweight analysis, but no explicit when-not.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fleet_evaluateA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses input acceptance (floats or Q0.16 integers, auto-conversion), output structure (nodes, episodes with fields, fleet_summary, credits), and behavior ('auto-converts per node'). No side effects or limitations are mentioned, but the core behavior is well-described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise yet information-rich, front-loading the purpose and covering input, output, and usage in three sentences with no redundancy. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (2 parameters, no output schema), the description sufficiently covers input format, output structure, conversion behavior, and usage context. It also distinguishes from the persistent alternative, making it complete for one-shot fleet evaluation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% but the description adds value by explaining the conversion semantics for scores, the episode length default, and the response structure. It also suggests chaining, going beyond schema constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Evaluate'), resource ('fleet of model nodes'), and purpose ('prediction stability'). It also distinguishes itself from siblings by explicitly mentioning 'For persistent multi-round fleet monitoring, use fleet_session_create instead.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool vs alternatives: it suggests chaining to 'visualize, alert_check, or compare_windows' and advises using 'fleet_session_create' for persistent multi-round monitoring.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fleet_session_createA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses return format (session ID and fields) and constraints. Without annotations, could mention idempotency or side effects, but current description is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences plus workflow line; no redundancy. Front-loaded with purpose and key info.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers all aspects: purpose, constraints, response, workflow. No output schema needed given detailed response description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (baseline 3). Description adds meaning by clarifying node_names must match node_count, which schema does not enforce.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'Create a new persistent fleet monitoring session' with specific verb-resource pair. Distinguishes from sibling tools like fleet_session_delete and fleet_session_state.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides full workflow: fleet_session_create → fleet_session_round (repeat) → fleet_session_state (check) → fleet_session_delete (cleanup). Also notes 'Max 16 nodes' constraint.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• healthA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes the response format (status, version, latency_ms) implying read-only, no side effects. Could explicitly state non-destructive nature, but sufficient for a health check.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences, front-loaded with purpose and response format. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter, no-output-schema tool, the description provides all necessary context: purpose, response format, and usage timing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so description need not add parameter info. Baseline 4 applies as schema coverage is 100% and no parameters require explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool checks CI-1T engine health with specific response fields. Distinguishes from sibling tools like evaluate and fleet_evaluate by being a pre-operation health check.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises calling before evaluate/fleet operations to verify engine reachability, providing clear usage context and when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• interpret_scoresA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description discloses response structure, input auto-detection, and that it makes no external calls. It does not mention any potential side effects or logging, but for a simple stateless tool this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely concise: two sentences cover purpose, behavior, input, output, and alternatives. Front-loaded with key info.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description fully documents the return value structure. It covers all needed context: what it does, what it returns, acceptable inputs, and alternative tool. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% but description adds value by explaining auto-detection of float vs integer, and details the response breakdown. This goes beyond just repeating the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Analyze' and the resource 'raw prediction scores with statistical breakdown'. It distinguishes itself from siblings like evaluate and convert_scores by specifying the response structure and input types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when to use this tool: for statistical breakdown, and when not: for full stability classification, use evaluate tool. Also notes no API call, no auth, no credits.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• onboardingA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description compensates by stating 'No auth required' and describing return content. Adds value beyond schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences with no wasted words, front-loaded with purpose and content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Fully describes purpose, usage, and output for a simple, parameterless tool. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Zero parameters, so baseline 4. Description mentions output contents, which is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it's a welcome guide for new users, listing specific contents (setup instructions, tools, pricing, links). Distinct from sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to call when someone is new or asks how to get started, providing clear usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.