Server Quality Checklist

Tool Scores

• get_consultation_historyC
  2.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description fails to disclose behavioral traits beyond the basic read operation. There is no mention of authentication requirements, rate limits, or data scope (e.g., whether it returns consultations for a specific user or all).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely short (3 words) but merely restates the tool name without adding value. It is under-specified rather than efficiently concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, no output schema, and no annotations, the description is severely incomplete. It does not explain the return format, filtering behavior, or how results are ordered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 4 parameters with 0% description coverage, and the tool description does not explain any of them. The agent has no guidance on how to use 'limit', 'since', 'advisor_type', or 'consultation_id'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'retrieve' and resource 'consultations', making the purpose understandable. It distinguishes from sibling tool 'consult', which likely handles creation or management.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like 'consult' or other siblings. The description provides no context on appropriate scenarios, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• athena_indexC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description must disclose behavioral traits. It only says 'Trigger', which is vague and does not indicate if the operation is destructive, asynchronous, or requires specific permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short and front-loaded, but it sacrifices essential detail for brevity. Every sentence earns its place, but more context is needed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, output schema, and parameter descriptions, the description is insufficient for correct tool selection and invocation. It omits critical behavioral and parameter details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description provides no meaning for the 'mode' or 'paths' parameters. The agent has no guidance on what values to use or their effects.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Trigger project memory indexing' uses a specific verb and resource, clearly indicating the action and target. However, it does not differentiate from sibling tools like athena_remember or athena_status, which could be ambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No usage guidelines are provided. The description does not specify when to use this tool versus alternatives, nor does it mention prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• athena_rememberC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It states 'persist' implying a write operation, but does not mention idempotency, overwrite behavior, or any side effects. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise but omits crucial details. It is not front-loaded with critical information, and every sentence should earn its place; here, the single sentence is too brief.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters (1 required) and no output schema or annotations, the description is insufficiently complete. It does not explain return values, error cases, or how parameters interact, leaving the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no explanation of parameters. The parameter names ('tags', 'affects', 'decision', 'rationale') provide some semantics, but the description fails to clarify their purpose or format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses a specific verb ('persist') and identifies the resource ('design decision' and 'project memory'). It clearly states the tool's action, distinguishing it from sibling tools like athena_index (search) and consult (questioning).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like consult or athena_index. The description does not mention contexts, prerequisites, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• consultC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior fully. It mentions advisor types but omits whether the tool is read-only, requires permissions, has rate limits, or what side effects exist. No information about session context or constraints is provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, consisting of one sentence and a list of advisor types. It avoids fluff but could benefit from a structured breakdown of key parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, nested objects, and no output schema, the description is too brief. It does not explain how to structure questions, what constraints modify, or what session_context does, leaving the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should compensate. It only explains 'advisor_type' by listing possible values. Other parameters (mode, question, constraints, session_context) are not described, leaving their semantics to the agent's inference.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool consults Athena for expert engineering advice, with a specific list of advisor types (architecture, security, debugging, planning, review, optimization, documentation). This distinguishes it from sibling tools like athena_index, athena_remember, athena_status, and get_consultation_history, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites or situations to avoid. It only hints at advisor types but lacks explicit usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• athena_statusA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behaviors; it only states it 'returns' status, implying a read operation, but provides no details on side effects, auth needs, or failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single concise sentence that gets straight to the point with no unnecessary words. Front-loaded with the verb and resource.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description is minimal. It covers the basic purpose but leaves ambiguity about what exactly 'status' includes (e.g., format, error conditions).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so the schema inherently provides full coverage. The description adds nothing beyond that, but per guidelines, baseline is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Return') and the specific resources ('Athena project memory and provider status'), distinguishing it from sibling tools like athena_index or athena_remember.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives (e.g., athena_index, consult). Lacks context about prerequisites or scenarios where this tool is preferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.