Server Quality Checklist

Tool Scores

• memory_get_userC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states it's a 'Get' operation, implying read-only behavior, but doesn't mention permissions, rate limits, pagination details (beyond implied by skip/take), or what 'collected memories' entails. This leaves significant gaps for a tool with 5 parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's front-loaded and appropriately sized for its function, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 parameters, no annotations, no output schema), the description is insufficient. It doesn't explain what 'memories' are, the return format, error conditions, or behavioral nuances like pagination limits. For a data retrieval tool with multiple parameters, more context is needed to guide effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional semantic context beyond implying filtering by user and memoryId, which is already covered in the schema. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('collected memories for a specific user'), making the purpose evident. However, it doesn't explicitly differentiate from sibling tools like 'memory_list' or 'user_get', which might have overlapping functionality, so it doesn't reach the highest score of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'memory_list' or 'user_get'. It lacks context about prerequisites, exclusions, or specific scenarios, leaving the agent with minimal usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• user_createC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. 'Create a new user' implies a write/mutation operation, but the description doesn't mention authentication requirements, permissions needed, whether the operation is idempotent, what happens on duplicate emails, or what the response contains. This leaves significant gaps for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point. It's appropriately sized for the tool's purpose with no wasted words or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a user creation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what happens after creation (e.g., does it return the user object? send a confirmation email?), what validation occurs, or error conditions. The mention of 'optional product access' is helpful but doesn't compensate for the broader context gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all parameters are documented in the schema. The description adds minimal value beyond the schema by mentioning 'optional product access' which corresponds to the 'products' parameter, but doesn't provide additional context about product IDs or how access is granted. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create a new user') and resource ('user'), and mentions optional product access which adds specificity. However, it doesn't explicitly differentiate from sibling tools like user_invite or user_update, which would require more precise language about when to use each.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like user_invite or user_update. It mentions 'optional product access' but doesn't explain when this feature is relevant or what prerequisites exist for user creation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• user_deleteC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool deletes a user but fails to mention critical details like whether this action is reversible, what permissions are required, what happens to associated data, or any rate limits. This leaves significant gaps for a destructive operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with zero wasted words, making it highly efficient and front-loaded. It immediately conveys the core action without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive tool with no annotations and no output schema, the description is inadequate. It lacks essential context such as the tool's impact, return values, error conditions, or how it differs from sibling tools, leaving the agent with insufficient information for safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents both parameters ('studio' and 'email') with their types, descriptions, and requirements. The description adds no additional parameter information beyond what the schema provides, which is acceptable but not additive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Delete') and the resource ('a user by email'), making the purpose immediately understandable. However, it doesn't differentiate this tool from sibling tools like 'doc_delete' or 'user_update' beyond the resource type, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'user_update' or 'doc_delete', nor does it mention prerequisites, permissions, or consequences. It simply states what the tool does without context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• user_getC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states it's a read operation ('Get'), implying non-destructive behavior, but doesn't mention authentication requirements, rate limits, error conditions, or what 'details' include. For a user lookup tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core purpose and appropriately sized for a simple lookup tool. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a user retrieval tool with no annotations and no output schema, the description is incomplete. It doesn't explain what 'details' include, what format they're returned in, or potential error cases. While the schema covers inputs well, the overall context lacks sufficient information for confident tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters (studio with default and enum values, email as required). The description adds no parameter-specific information beyond what's in the schema - it mentions 'by email' which is already covered. Baseline 3 is appropriate when schema does all the work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('details for a specific user'), specifying retrieval by email. It distinguishes from siblings like user_list (list all users) and user_create/update/delete (write operations), though it doesn't explicitly name these alternatives. The purpose is specific but lacks explicit sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose user_get over user_list for a single user, or when to use memory_get_user for user-related memory retrieval. There are no prerequisites, exclusions, or contextual usage hints provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• user_inviteC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While it indicates this sends emails (an external action) and grants product access (a permission change), it doesn't disclose important behavioral aspects like whether this requires admin permissions, what happens if emails are invalid, whether invitations can be resent, rate limits, or what the expected response looks like. For a tool with mutation capabilities and no annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that communicates the core functionality without unnecessary words. It's front-loaded with the main action and includes the key qualification about optional product access. Every word earns its place in this concise formulation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that performs user invitations (a mutation with external email effects) with no annotations and no output schema, the description is incomplete. It doesn't explain what happens after invocation - whether users receive immediate emails, what the invitation contains, whether there are confirmation steps, or what the tool returns. The description should provide more context about this non-trivial operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters thoroughly. The description adds minimal value beyond the schema - it mentions 'optional product access' which aligns with the optional productIds parameter, but doesn't provide additional context about parameter interactions, default behaviors, or usage patterns. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Send email invitations') and target ('to new users'), with additional context about optional product access. It distinguishes from siblings like user_create or user_update by focusing specifically on invitation functionality rather than user record management. However, it doesn't explicitly differentiate from all sibling tools like user_get or user_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like user_create (which might create users without invitations) or user_update (which might modify existing users). There's no mention of prerequisites, limitations, or typical scenarios for sending invitations versus other user management operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• user_updateC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states it's an update operation but doesn't cover critical aspects like required permissions, whether changes are reversible, rate limits, or what the response looks like (since there's no output schema). This is a significant gap for a mutation tool with multiple parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a mutation tool with 7 parameters, no annotations, and no output schema), the description is incomplete. It lacks behavioral context, usage guidelines, and details on return values, which are essential for an agent to use this tool effectively without trial and error.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so parameters are well-documented in the schema itself. The description adds minimal value beyond implying that 'details, products, or usage' correspond to the parameters, but it doesn't provide additional context like parameter interactions or default behaviors. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Update') and resource ('existing user's details, products, or usage'), making the purpose understandable. However, it doesn't explicitly differentiate from sibling tools like 'user_create' or 'user_get', which would require mentioning it's for modifying existing users rather than creating new ones or retrieving data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., user must exist), exclusions (e.g., not for creating users), or comparisons to siblings like 'user_create' or 'user_get', leaving the agent to infer usage from context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• doc_getB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While 'Retrieve' implies a read-only operation, it doesn't specify whether this requires authentication, has rate limits, returns structured data, or handles errors. For a tool with zero annotation coverage, this leaves significant behavioral gaps unaddressed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that directly states the tool's purpose without any unnecessary words or structural fluff. It's front-loaded with the core action and resource, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple retrieval tool with two parameters (fully documented in the schema) and no output schema, the description is minimally adequate. However, it lacks context about authentication needs, error handling, or return format, which would be helpful given the absence of annotations and output schema. It meets basic requirements but leaves room for improvement.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions retrieving by 'ID', which aligns with the 'documentId' parameter in the schema. However, with 100% schema description coverage, the schema already fully documents both parameters (including the 'studio' parameter with its default value and options). The description adds minimal value beyond what's already in the structured schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Retrieve') and resource ('a specific document by ID'), making the purpose immediately understandable. However, it doesn't differentiate this tool from sibling tools like 'doc_get' vs 'doc_list' or 'user_get', which would require explicit comparison to achieve a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'doc_list' (for listing documents) or 'user_get' (for retrieving user data). There's no mention of prerequisites, context requirements, or any explicit 'when' or 'when not' instructions, leaving usage decisions entirely to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• doc_listB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions pagination, which is useful, but doesn't describe what 'documents' contain, the return format, error conditions, rate limits, authentication requirements, or whether this is a read-only operation. For a list tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that communicates the core functionality without unnecessary words. It's appropriately sized for a list operation and front-loads the essential information. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (list operation with pagination), no annotations, and no output schema, the description is minimally adequate but incomplete. It covers the basic action and mentions pagination, but doesn't address what constitutes a 'document', return format, error handling, or relationship to sibling tools. For a tool with no structured behavioral hints, it should provide more context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents all three parameters. The description adds no additional parameter information beyond mentioning 'pagination' generally, which aligns with skip/take parameters but provides no extra semantic context. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List all documents') and resource ('in the Pickaxe studio'), making the purpose immediately understandable. It distinguishes from siblings like doc_get (retrieve specific document) and doc_create/delete (mutations). However, it doesn't explicitly differentiate from other list operations like studios_list or products_list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose doc_list over doc_get for retrieving documents, or how it relates to other list operations like studios_list. There's no context about prerequisites, permissions, or typical use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• memory_listB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action ('List') but fails to describe key behaviors such as pagination mechanics (implied by 'skip' and 'take' parameters), return format, error conditions, or authentication requirements. This leaves significant gaps for a tool with three parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded with the core action and resource, making it easy to parse and understand immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list operation with no annotations and no output schema, the description is minimally adequate. It specifies the resource ('memory schemas') and scope ('in the studio'), but lacks details on output structure, pagination behavior, or error handling. Given the tool's moderate complexity (three parameters), it should provide more context to be fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema fully documents all three parameters. The description adds no additional parameter semantics beyond what's in the schema, but since the schema is comprehensive, a baseline score of 3 is appropriate as the description doesn't need to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all memory schemas defined in the studio'), making the purpose unambiguous. It doesn't explicitly differentiate from sibling tools like 'memory_get_user' or 'doc_list', but the specificity of 'memory schemas' provides some implicit distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'memory_get_user' or 'doc_list'. It lacks any mention of prerequisites, exclusions, or contextual triggers for selection, leaving the agent to infer usage based solely on the tool name and description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• products_listB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is a list operation but doesn't describe what 'list all' entails (e.g., pagination, sorting, filtering capabilities), whether it requires authentication, or what format the output takes. The description is minimal and lacks important behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point with zero wasted words. It's appropriately sized for a simple list operation and is perfectly front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one optional parameter and no output schema, the description is minimally adequate. However, without annotations and with no output schema, it should ideally provide more context about what the tool returns and any behavioral constraints. The description covers the basic purpose but leaves gaps in usage and behavioral understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the single parameter 'studio' fully documented in the schema. The description doesn't add any parameter-specific information beyond what's already in the schema, so it meets the baseline expectation without providing extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('products/bundles in the Pickaxe studio'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'doc_list' or 'studios_list' beyond mentioning the specific resource type.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, appropriate contexts, or compare it to similar sibling tools like 'doc_list' or 'studios_list' that also list resources.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chat_historyB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions the tool fetches history and suggests use cases, it doesn't describe important behavioral aspects: whether this requires authentication, rate limits, what the return format looks like (beyond the 'format' parameter), pagination behavior beyond skip/limit parameters, or error conditions. For a tool with 5 parameters and no annotation coverage, this leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that efficiently state the core purpose and three use cases. It's front-loaded with the main action ('Fetch conversation history') and wastes no words. However, the second sentence could be slightly more structured by separating the use cases with semicolons or bullet points for even clearer scanning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters, no annotations, and no output schema, the description provides adequate but incomplete context. It clearly states what the tool does and suggests use cases, but doesn't address behavioral aspects like authentication needs, rate limits, error handling, or what the return data structure looks like. For a data retrieval tool with multiple parameters, more complete behavioral context would be helpful despite the good schema documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with all parameters well-documented in the schema itself. The description doesn't add any meaningful parameter semantics beyond what's already in the schema - it doesn't explain parameter relationships, provide examples, or clarify edge cases. With complete schema coverage, the baseline score of 3 is appropriate as the description doesn't compensate but also doesn't need to given the comprehensive schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Fetch conversation history') and resource ('for a Pickaxe agent'), making the purpose immediately understandable. It distinguishes this tool from sibling tools (which are all about docs, memory, products, studios, or users) by focusing on chat history. However, it doesn't explicitly differentiate from potential non-existent siblings like 'chat_analytics' or 'chat_summary'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage context by listing three use cases: 'analyze user questions, identify KB gaps, and review agent performance.' This gives some guidance on when this tool would be appropriate. However, it doesn't explicitly state when NOT to use it or mention alternatives among the sibling tools (none of which appear to be direct alternatives for fetching chat history).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• doc_deleteB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool deletes and removes from all agents, which implies a destructive, irreversible action, but doesn't mention permissions needed, error conditions, confirmation requirements, or what happens to associated data. For a deletion tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with zero waste. The first sentence states the core action, and the second adds critical scope information. Every word earns its place, and it's front-loaded with the primary purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given this is a destructive deletion tool with no annotations and no output schema, the description is incomplete. It doesn't address permissions, side effects, error handling, or return values. For a high-stakes operation like deletion, more contextual information is needed to guide safe usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents both parameters (studio and documentId). The description adds no additional parameter information beyond what's in the schema, such as format details for documentId or implications of studio choice. Baseline 3 is appropriate when the schema does all the work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Delete') and resource ('a document from Pickaxe'), with additional scope information ('removes it from all connected agents') that distinguishes it from potential siblings like doc_disconnect. It provides a complete picture of what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like doc_disconnect (which might remove connections without deletion) or user_delete (for different resources). It mentions the effect but not the appropriate context or prerequisites for invoking this destructive operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• doc_disconnectB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the action is to 'disconnect/unlink' and remove from knowledge base, implying a mutation, but doesn't disclose behavioral traits like whether this is reversible, requires specific permissions, has side effects on the document or agent, or what the response looks like (e.g., success/failure).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action and effect. Every word earns its place, with no redundancy or unnecessary details, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a mutation tool with no annotations and no output schema), the description is incomplete. It lacks information on behavioral context (e.g., reversibility, permissions), response format, and usage guidelines, leaving gaps for an AI agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (studio, documentId, pickaxeId) with their types and descriptions. The description doesn't add any meaning beyond what the schema provides, such as explaining relationships between parameters (e.g., documentId and pickaxeId must correspond to an existing link).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('disconnect/unlink'), the resource ('a document from a Pickaxe agent'), and the effect ('removing it from the agent's knowledge base'). It distinguishes from siblings like doc_connect (linking) and doc_delete (deleting the document entirely).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., the document must be linked first), exclusions, or comparisons to siblings like doc_delete (which might remove the document entirely versus just unlinking).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• doc_createB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the tool creates documents but does not disclose behavioral traits such as required permissions, whether creation is idempotent, error handling, or rate limits. The mention of content sources adds minimal context, but critical operational details are missing for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and followed by a concise detail on content sources. Every sentence adds value without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is minimally adequate for a creation tool but lacks depth. It covers the what and how (content sources) but omits behavioral aspects like permissions, side effects, or response format. For a 4-parameter mutation tool, more context would be needed for a higher score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing clear documentation for all parameters. The description adds value by clarifying the mutual exclusivity of rawContent and website ('Use this OR rawContent, not both'), which is not in the schema. However, it does not elaborate on parameter interactions or provide additional semantic context beyond the schema's basics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create a new document') and resource ('in Pickaxe knowledge base'), with additional detail about content sources ('from raw content or scrape a website URL'). It distinguishes from siblings like doc_delete (deletion) and doc_get (retrieval), though not explicitly. A 5 would require explicit sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for document creation, but lacks explicit guidance on when to use this tool versus alternatives like doc_connect or doc_list. It mentions content source options (rawContent or website), which provides some context, but does not specify prerequisites, constraints, or scenarios where other tools might be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• doc_connectA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states this is a mutation operation ('connect/link'), implying it modifies data, but does not disclose behavioral traits such as required permissions, whether the connection is reversible, error handling, or rate limits. The description adds minimal context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action and purpose without unnecessary words. Every part earns its place by specifying the action, resource, target, and outcome, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (mutation with 3 parameters), no annotations, and no output schema, the description is minimally adequate. It covers the basic purpose but lacks details on behavior, output, or error handling, leaving gaps that could hinder an agent's ability to use it correctly in all contexts.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (studio, documentId, pickaxeId) with descriptions and defaults. The description does not add any meaning beyond what the schema provides, such as explaining parameter interactions or constraints, so it meets the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('connect/link'), resource ('document'), and target ('Pickaxe agent'), distinguishing it from siblings like doc_disconnect (opposite action) and doc_create/delete (different document operations). It explicitly mentions adding to the agent's knowledge base, which clarifies the purpose beyond just linking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying the action and target, but does not explicitly state when to use this tool versus alternatives like doc_disconnect or prerequisites. It mentions adding to a knowledge base, which provides some context, but lacks explicit guidance on scenarios or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• studios_listA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states it's a list operation, implying read-only behavior, but does not cover aspects like pagination, rate limits, authentication requirements, or error handling. This is a significant gap for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the main action ('List all configured Pickaxe studios') and adds a useful detail ('and the current default'). There is no wasted verbiage, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema), the description is adequate but has clear gaps. It explains what the tool does but lacks behavioral context (e.g., how results are formatted, any limitations). Without annotations or an output schema, the description should do more to compensate, but it meets the minimum viable threshold.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100% (as there are no parameters to describe). The description does not need to add parameter details, so a baseline score of 4 is appropriate, as it efficiently avoids redundancy.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all configured Pickaxe studios') and the resource ('studios'), and includes an additional output detail ('and the current default'). It distinguishes itself from sibling tools like 'doc_list' or 'user_list' by focusing specifically on studios.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or comparisons to sibling tools, leaving the agent to infer usage based on the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• user_listA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states what the tool does, not behavioral traits. It doesn't disclose whether this is a read-only operation, pagination behavior beyond skip/take parameters, rate limits, authentication needs, or what format the 'usage stats' will be in.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without any wasted words. Every element (verb, resource, attributes) earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list tool with 3 parameters and 100% schema coverage but no annotations or output schema, the description adequately states the purpose but lacks behavioral context. It doesn't explain what 'product access and usage stats' includes or the response format, leaving gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (studio, skip, take) with their types and defaults. The description adds no additional parameter semantics beyond what's in the schema, maintaining the baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all users in the Pickaxe studio') with specific attributes ('product access and usage stats'). It distinguishes from siblings like user_get (single user) and user_create/delete/update (mutations).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for listing users with stats, but provides no explicit guidance on when to use this vs. alternatives like user_get for single users or memory_list for memory data. No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.