Server Quality Checklist

Tool Scores

• get_study_statusD
  1/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Tool has no description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness1/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Tool has no description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool has no description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Tool has no description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose1/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Tool has no description.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Tool has no description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_study_resultsC
  2.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It partially succeeds by revealing the result structure contains 'themes' with 'quotes fields', hinting at a nested qualitative data format. However, it omits critical operational details: whether this is read-only (implied by 'Returns' but not confirmed), caching behavior, result size limits, or latency characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief (two sentences), but awkwardly mixes concerns: the first states functionality tersely ('Returns analysis results'), while the second jumps to post-invocation presentation guidelines. This structural confusion places output formatting instructions in a field meant for capability description, creating a minor organizational misfit despite brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description partially compensates by describing the result contents (themes containing quote fields), which helps the agent understand what data structure to expect. However, with zero annotations and zero parameter schema coverage, significant gaps remain for a complete invocation context, particularly regarding the optional 'format' parameter's impact on result granularity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description text compensates not at all. Neither 'study_id' (despite being a required UUID) nor 'format' (despite having enum values 'summary'/'full' with unclear semantic differences) are mentioned in the text. The agent must rely solely on parameter names, which is insufficient for the 'format' option distinctions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it 'Returns analysis results' which is vague, though the mention of 'participant responses', 'themes', and 'quotes field' provides domain context about qualitative research data. However, it fails to distinguish clearly from sibling 'get_study_status' (which likely returns metadata vs. actual results), leaving ambiguity about which tool retrieves substantive data versus operational status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance is provided on when to use this tool versus alternatives like 'get_study_status' or 'create_study'. The second sentence provides presentation instructions ('always quote specific participant responses verbatim') rather than invocation guidelines, failing to clarify prerequisites or selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_studyB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. States it's an update operation but lacks critical behavioral details: partial vs full replacement, validation rules, error handling for non-existent studies, or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. Front-loaded with main action ('Updates'), followed by specific use cases. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no annotations, no output schema, and 50% param coverage, gaps remain. Description covers primary use cases but doesn't compensate for missing behavioral details or undocumented parameters. Adequate but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50% (target_interviews and study_media have descriptions; study_id and is_link_disabled do not). Description maps use cases to the two documented parameters but fails to explain the undocumented boolean 'is_link_disabled' or required 'study_id'. Baseline 3 appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb ('Updates') and resource ('existing study'). Lists specific use cases (interview slots, media). Distinguishes from siblings implicitly (update vs create/delete/get), though 'modify interview guide' doesn't map clearly to any schema parameter.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides specific usage scenarios ('Use this to increase interview slots...'), but lacks explicit differentiation from siblings (e.g., when to use create_study vs this) and no guidance on when not to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_studyA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full disclosure burden. Successfully discloses return value ('interview_link'), side effect ('Starts with 1 interview slot'), and optional media behavior. Missing: auth requirements, rate limits, or error conditions (e.g., duplicate study names).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences. First establishes core operation and return value; second covers default slot count and key optional feature. No redundant phrases or tautology despite null title.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Explains return value (interview_link) compensating for missing output schema, and covers the nested study_media object. However, for 7 parameters with 86% lacking schema descriptions, description omits explanation of required parameters' semantics and relationships between context / research goals.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 14% (only study_media described at top level). Description mentions study_media functionality but fails to compensate for 6 undocumented parameters including required fields (key_research_goal, business_context). Does not explain what content belongs in research goals vs business context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Specific verb ('Creates') + specific resource ('user interview study') clearly identifies the operation. Implicitly distinguishes from siblings delete_study/update_study through the creation verb, and from get_* tools by specifying it returns an interview_link.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies creation use case through verb choice and mentions it 'Starts with 1 interview slot,' suggesting initialization behavior. However, lacks explicit guidance on when to use update_study instead, or prerequisites for creation (e.g., workspace requirements).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_studyA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full behavioral disclosure burden. It succeeds by warning of irreversibility ('Permanently'), scope of destruction ('all associated data'), and economic side effects ('Releases unused reserved credits'). Deducted one point for not indicating failure behavior (e.g., what happens if study_id doesn't exist).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. Critical information ('Permanently') front-loaded. First sentence covers the core operation; second sentence covers the credit side effect. No redundancy or generic filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers the essential risks (destruction, credits) for a destructive tool with no output schema. However, missing error handling patterns (404 behavior), authorization requirements, or confirmation requirements prevents a higher score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, requiring the description to compensate. It fails to mention the study_id parameter, its expected format (UUID), or how to obtain it, even though this is the sole parameter. While somewhat inferable from the tool name, the description adds zero semantic value beyond the schema structure.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Excellent specificity: 'Permanently deletes' provides a precise verb, 'study' identifies the resource, and 'all associated data' clarifies scope. The description clearly distinguishes this from sibling tools like update_study (modification vs destruction) and get_* (read-only).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The word 'Permanently' implicitly signals this is for complete removal, not modification (update_study) or creation (create_study). However, it lacks explicit guidance on when to prefer deletion over archival, or prerequisites like 'study must be in terminal state'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.