Server Quality Checklist

Tool Scores

• list_coursesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description goes beyond this by disclosing pagination behavior (single page, has_more/next_cursor), the local failure mode when the OAuth scope is missing, and the case-insensitive partial match for filter_title. This adds meaningful behavioral context without contradicting any annotation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. It leads with the purpose, follows with a critical pagination warning, then parameter detail, and finally the scope requirement. Each sentence earns its place with no redundancy. The formatting uses line breaks for readability, making it easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present and annotations covering the safety profile, the description covers the essential operational aspects: pagination handling, filter behavior, and authentication scope. Minor omissions such as explicit page_size effect and sort order do not prevent correct usage. Overall, it is sufficiently complete for an agent to call this tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains filter_title as a case-insensitive partial match, which is valuable. It also implies page_cursor usage via the pagination discussion ('call again with next_cursor'), but does not explicitly link page_cursor to next_cursor or define the effect of page_size. The description partially compensates but leaves page_size semantics undocumented, so a 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific statement - 'List the organization's non-deleted, non-draft courses' - which clearly identifies the action, resource, and filtering criteria. This is distinct from sibling tools (check_access, describe_capabilities, report_a_problem) which all serve different purposes. No ambiguity remains about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on pagination: it returns one page, instructs the agent to check `has_more` and call again with `next_cursor` before drawing conclusions. It also states the OAuth scope requirement. It does not mention alternatives, but given the unrelated sibling tools, such exclusions are unnecessary. This qualifies as clear context without exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_accessA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and non-destructive, but the description adds substantial context beyond them: it needs no credential, makes no Skilljar call when nothing is configured, works even when everything else fails, and returns no secret material—only credential status and granted scopes. This fully discloses behavior in way annotations cannot.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is relatively long but every sentence contributes to operational understanding. It opens with the core purpose, then adds usage guidance and edge-case behavior. While it could be tightened slightly, the structure is logical and information-dense without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the zero-parameter signature, rich annotations, and presence of an output schema, the description covers all necessary context: what it does, when to use it, how to act on results, its independence from credentials, and its reliability even when other tools fail. Nothing important is missing for correct invocation and interpretation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so baseline is 4. The description adds no parameter-specific meaning because there are none. It correctly notes that it requires no credential itself, which is about invocation context rather than parameters. Baseline applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: it reports which Skilljar credential is configured and working and what each unlocks. This is specific and distinguishes it from sibling tools like describe_capabilities and list_courses, though it does not explicitly name those alternatives. The 'call this first' instruction further clarifies its diagnostic role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Call this first whenever a tool reports a credential problem' and instructs the agent to relay the result rather than retry, since retries fail identically. It also explains the two-independent-credentials scenario, which helps decide when to act. This is strong usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_capabilitiesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering the safety profile. The description adds behavioral context beyond that: it explains the distinction between existing capabilities and enabled ones (available_but_disabled field), and states that the policy is immutable from this environment. This is valuable context not present in the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. It front-loads the core purpose, then immediately provides a usage directive, highlights the key field, and closes with a policy note. Every sentence carries meaningful information without redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with an output schema and annotations covering safety, the description is complete. It tells the agent what the tool does, when to call it (after refusal), how to interpret the critical output field, and what limitations exist (immutability). An agent would have no gaps in knowing when and how to use this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema provides full coverage and the description correctly does not attempt to document parameters. The description focuses on the tool's behavior and output interpretation, which is appropriate. Baseline for 0 params is 4, and nothing here warrants a downgrade.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear purpose: describing what the install is permitted to do and what it could do if reconfigured. It uses a specific verb ('describe') and resource ('capabilities'), and distinguishes itself from siblings by focusing on permission introspection rather than access checks, problem reporting, or course listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage guidance: 'Call this after a refusal.' It also explains how to interpret the results, specifically telling the agent to inform the user of the setting to change when a capability is listed as available_but_disabled. It does not mention alternatives, but the context is clear and no alternative is needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• report_a_problemA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint. The description adds significant behavioral context: it states that the report carries version, platform, active policy, and excludes Skilljar ids and credential values, and that user-provided text is reproduced verbatim. This goes beyond the structured data without contradicting it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. The primary action is front-loaded in the first sentence, followed by usage guidance and behavioral details. Each sentence adds value, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, non-destructive, read-only), the description fully covers when to use it, what it does, and what the output contains and excludes. The output schema exists, so the description need not detail return values. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It does: 'Put what you actually observed in `what_happened`; that text is reproduced verbatim.' This gives the parameter a clear semantic role and behavior, which the bare schema lacks.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Assemble a bug report about this server for the user to file.' It clearly distinguishes the tool from siblings like check_access or list_courses, which serve different purposes. The action is concrete and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use conditions are given: 'Call this when a tool is missing, a result contradicts its own description, or an error makes no sense - and when the user asks how to report something.' This tells the agent precisely when to invoke it, leaving no ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.