Server Quality Checklist

Tool Scores

• statusC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions operations but doesn't clarify whether these are read-only or mutating operations (e.g., 'generate_embeddings' and 'run_sync' sound like mutating operations). No information about permissions, side effects, rate limits, or what happens when operations fail is provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise but could be better structured. It starts with a vague phrase 'System health, sync status, and management operations' then lists operations. The information is front-loaded but the initial phrase doesn't clearly communicate the tool's purpose. Some sentences could be more efficiently worded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 distinct operations (some potentially mutating), no annotations, and no output schema, the description is insufficient. It doesn't explain what each operation returns, what 'sync from Otter.ai' entails, what 'pending jobs' means for generate_embeddings, or any error conditions. The description leaves too many behavioral questions unanswered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description doesn't add meaningful parameter semantics beyond what's in the schema - it lists operation names but doesn't explain them further. The baseline of 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description lists multiple operations (health, sync, stats, generate_embeddings, run_sync) but doesn't clearly state a unified purpose. It reads more like a menu of options rather than a coherent tool purpose. While it distinguishes from sibling tools 'meetings' and 'search' by focusing on system operations, it lacks a clear overarching verb+resource statement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use each operation: 'Use health to check system state, sync to see last sync info, stats for database statistics, generate_embeddings to process pending jobs, run_sync to sync from Otter.ai.' This gives clear context for each option, though it doesn't explicitly state when NOT to use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• meetingsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it lists operations, it doesn't describe important behavioral traits like whether this is a read-only operation, authentication requirements, rate limits, pagination behavior beyond the schema's offset/limit, error handling, or what 'stats' returns. For a tool with 5 parameters and multiple operations, this leaves significant gaps in understanding how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that efficiently communicate the tool's purpose and operations. The structure is front-loaded with the core purpose followed by operational details. Every sentence earns its place, though the second sentence could be slightly more structured for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, 4 distinct operations, no annotations, no output schema), the description is incomplete. It doesn't explain what the tool returns for different operations, how results are structured, error conditions, or behavioral constraints. For a multi-operation tool with no output schema, the description should provide more context about expected outputs and operational boundaries.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value beyond the schema by mentioning the operations that correspond to the 'operation' enum values, but doesn't provide additional semantic context about parameter interactions or usage patterns. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Browse and retrieve meeting transcripts from Otter.ai' with specific operations listed (list, get, recent, stats). It distinguishes itself from the 'search' sibling tool by focusing on browsing/retrieving rather than searching, though the distinction could be more explicit. The verb+resource combination is specific and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the listed operations but doesn't explicitly state when to use this tool versus the 'search' sibling tool. It provides operational context (list for all, get for specific, recent for last N days, stats for database info) but lacks clear guidance on tool selection criteria or exclusion scenarios. The implied usage is helpful but not comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the tool's behavior by specifying it returns 'matching chunks with source citations' and describes the three search modes. However, it doesn't mention important behavioral aspects like rate limits, authentication requirements, error conditions, or pagination behavior for large result sets.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly sized at two sentences that each earn their place. The first sentence establishes the core functionality with three search modes, and the second sentence specifies the return format. No wasted words, well-structured, and front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (7 parameters, 3 search modes) and no output schema, the description provides adequate but incomplete coverage. It explains what the tool does and the return format, but doesn't address error handling, performance characteristics, or detailed output structure. With no annotations, more behavioral context would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 7 parameters thoroughly. The description adds minimal value beyond the schema by mentioning the three search modes (which are already in the enum) and the 'context_chunks' parameter's purpose. This meets the baseline expectation when schema coverage is high.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Search') and resource ('meeting transcripts'), specifies three distinct search modes (semantic, keyword, context), and distinguishes from sibling tools like 'meetings' and 'status' by focusing on transcript search functionality rather than meeting management or system status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use different search modes (semantic for AI similarity, keyword for exact match, context for surrounding transcript), but doesn't explicitly state when NOT to use this tool or mention alternatives among sibling tools. The guidance is helpful but not comprehensive about exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.