Server Quality Checklist

Tool Scores

• generate_challengeB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Lacks annotations; description gives minimal behavioral info (skill calibration), but omits details like whether it modifies state, requires auth, or what side effects exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single, front-loaded sentence with no redundancy; every word contributes value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema; description fails to specify the format or structure of the generated challenge, leaving the agent uncertain about return value.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%; description doesn't add meaning beyond schema, which already describes all three parameters adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates a JavaScript coding challenge based on topic or code, and distinguishes it from sibling tools like bookmark_concept or explain_concept.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage for generating practice challenges, but no explicit guidance on when to use vs alternatives or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• bookmark_conceptA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must cover behavioral traits. It only says 'Save' implying mutation but does not disclose any consequences, limits, or persistence details. For a mutation tool, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence with no wasted words. It efficiently conveys the tool's purpose and usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the description covers purpose and usage, it lacks behavioral transparency (e.g., what happens to existing bookmarks) and does not address the return value (no output schema). For a simple tool, it is adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (all 4 parameters have descriptions), so the baseline is 3. The tool description does not add any extra meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Save a JavaScript concept to study later' with a specific verb and resource, and it distinguishes from sibling tools (explain_concept, generate_challenge, get_learner_progress) which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear usage context: 'when you encounter something you want to review or don't fully understand yet.' Although it does not explicitly mention when not to use or name alternatives, the context is sufficient given the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_conceptA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses that the explanation is tailored, returns examples and links, and uses Kadmia progress. No mention of side effects, permissions, or state changes, but for a read-only explanation tool this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, no redundancy. Front-loaded with purpose, directly followed by outcome.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description explains return (explanation with examples and links). Sufficient for a simple tool, though could detail how depth and context parameters influence output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers all parameters 100%. Description adds value by explaining personalization based on skill level (not in schema) and hinting at context usage. Depth enum is clear in schema, so no extra needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to explain a concept tailored to the user's skill level and progress in Kadmia, with examples and links. It distinguishes from siblings (bookmark, challenge, progress check).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (when needing an explanation), but lacks explicit guidance on when not to use or alternatives. It does not mention that it is for learning contexts or that it personalizes based on progress, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_learner_progressA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It discloses the authentication requirement but does not mention whether the operation is read-only, any side effects, rate limits, or the format of returned data. This is adequate but leaves some behavioral details unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently conveys the purpose and a key requirement (authentication). No unnecessary information, front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no inputs and no output schema, the description covers the essential: what it does and the authentication prerequisite. It could be more complete by hinting at what the progress data includes, but for a straightforward retrieval tool, it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, so schema coverage is 100%. The description correctly does not add parameter information because none exist. Baseline score of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the purpose: 'Get your learning progress from Kadmia'. This is a specific verb ('Get') and resource ('learning progress'), and it distinguishes from sibling tools (bookmark, explain, generate) by focusing on retrieval of progress.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the authentication requirement via KADMIA_LEARNER_ID env var, providing clear context for when to use this tool. It does not explicitly state when not to use it or offer alternatives, but the sibling tools cover other actions, so usage is well-understood.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.