Server Quality Checklist

Tool Scores

• create_study_taskC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It does not mention the idempotency_key parameter's behavior, duplicate handling, return value, or side effects. The only non-essential detail is the date format requirement, which is not behavioral.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short, direct, and front-loaded with the core action. It has no redundant words, but it is under-specified. It earns a 4 for efficiency, though not a 5 because it omits essential context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters, 0% schema coverage, and no annotations, the description is insufficiently complete. It only covers due_at and ignores idempotency_key and other behavioral aspects. An output schema exists, which may cover return values, but the input semantics remain incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds value for due_at by specifying ISO 8601 format, but leaves title, description, and especially idempotency_key unexplained. The idempotency_key parameter is non-obvious and requires semantic clarification.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Create a study task' uses a specific verb and resource, clearly distinguishing it from sibling tools that search, get, list, or update. The purpose is unequivocal.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description simply states the action without any contextual cues, exclusions, or references to siblings like update_study_task_status. The ISO 8601 hint is parameter-specific, not usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_materialsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It only mentions the ranking algorithm and does not disclose whether the operation is read-only, what kind of materials are searched, or any side effects. Minimal behavioral context is given.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the core purpose. Every word adds value, and the algorithmic detail is relevant to distinguishing the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists, so return value structure is covered elsewhere. However, the description lacks guidance on when to use search versus get_document and does not clarify parameter semantics, leaving notable gaps for such a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not explain 'query' or 'limit'. The agent must infer their meanings from the tool name and context, which is insufficient for correct parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches course materials, and the mention of 'dense plus lexical reciprocal-rank fusion' distinguishes it from sibling tools like get_document. It is specific about both the action and the resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied by the search verb, but there is no explicit guidance on when to use this tool versus alternatives like get_document. No exclusions or alternative references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_study_task_statusB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It only states what statuses can be set, which is already in the schema enum. It does not mention what the tool returns, whether it is idempotent, what happens on invalid task_id, or any side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with zero wasted words. It conveys the action, target, and allowed values efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only 2 parameters and an output schema, so return values do not need explanation. However, for an update operation with no annotations, the description is minimal and lacks context about expected behavior on success/failure or prerequisites, leaving it adequate but with gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It repeats the status enum values but does not explain task_id beyond its name. The parameter names are self-explanatory, but the description adds no further semantic meaning, so it fails to compensate for missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The verb 'Set' clearly indicates the action, and 'study task status' identifies the resource. It further specifies the allowed values (pending, in_progress, completed), which distinguishes it from sibling tools like create_study_task or list_study_tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied by the description: you would use this when you need to change the status of a study task. However, there is no explicit guidance about when not to use it or how it compares to alternatives like create_study_task, so it stops at implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_documentA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It conveys that this is a read operation (non-destructive), which is important context, but it does not disclose what happens if the document is not found, whether special permissions are required, or any other side effects. The description adds some behavioral context but leaves gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It immediately states the action and the key parameter, making it highly efficient and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the availability of an output schema, the description does not need to explain return values. For a simple get-by-ID tool, the description is adequate but lacks guidance on how to obtain the document_id or what distinguishes 'complete' documents. It does not reference sibling tools, reducing its completeness in a broader context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the single parameter document_id. It adds minimal meaning by indicating the parameter is the document's identifier, but it does not explain where to obtain this ID, the expected format, or any constraints. This is insufficient for a parameter with zero schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Read') with a clear resource ('complete course document') and method ('using its document ID'). This clearly distinguishes it from sibling tools like search_materials (search) and create_study_task (create), making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a document ID and need the full content, but it does not explicitly state when not to use it or mention alternative tools like search_materials for locating documents. There is a clear context but no exclusions or explicit alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_study_tasksA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden but only reveals the optional status filter. It does not detail behavior such as default listing all tasks, sorting, or side effects, though the verb 'List' implies a read-only operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the core action and filter option without unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list operation with one optional parameter and an output schema, the description is sufficiently complete, though it could have mentioned default behavior like returning all tasks when status is not provided.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% coverage, so the description adds meaning by explaining that the 'status' parameter filters by the task's current status and is optional, which goes beyond the schema's bare parameter definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'List' with the resource 'study tasks' and states the optional filter by status, clearly distinguishing it from sibling tools like create_study_task and update_study_task_status, which are mutations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the verb 'List' and context, but does not explicitly state when to use this tool over alternatives or mention exclusions. However, sibling names make the distinction obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.