Server Quality Checklist

Tool Scores

• arango_queryC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint: false, implying potential mutations, but the description doesn't disclose behavioral traits beyond this. It doesn't explain that AQL queries can be read-only or include writes, what permissions are needed, potential side effects, or error handling. With annotations covering only the read/write hint, the description adds minimal context, leaving significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with a single sentence, 'Execute an AQL query', which is front-loaded and wastes no words. Every part of the sentence is necessary to convey the core purpose, making it efficient and well-structured for its brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of executing arbitrary AQL queries, the lack of output schema, and minimal annotations, the description is incomplete. It doesn't explain return values, error cases, or how results are structured, which is critical for a query tool. With no output schema and only basic annotations, the description should provide more context to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear documentation for 'query' and 'bindVars' parameters. The description adds no additional meaning beyond what the schema provides, such as examples of AQL syntax or how bindVars are used. Since the schema does the heavy lifting, the baseline score of 3 is appropriate, as the description doesn't compensate but also doesn't detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Execute an AQL query' states the action (execute) and resource (AQL query), making the purpose clear. However, it doesn't differentiate from sibling tools like arango_insert or arango_update, which also execute database operations but with different intents. The description is vague about what type of execution this involves (e.g., read vs. write queries).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention that this is for arbitrary AQL queries, while siblings like arango_insert or arango_update are for specific operations, or that it might be preferred for complex queries. There are no explicit when/when-not instructions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arango_updateB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare destructiveHint=true, indicating this is a mutation operation. The description adds minimal context by specifying it updates a document, but doesn't elaborate on behavioral traits like whether updates are partial/complete, if it returns the updated document, error handling, or permissions required. With annotations covering the destructive nature, it earns a baseline score for not contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core action, making it easy to parse quickly, which is ideal for conciseness in tool descriptions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (mutation with 3 params, no output schema) and annotations only covering destructiveness, the description is incomplete. It lacks details on return values, error cases, or how it differs from siblings, leaving gaps for an agent to understand full usage context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all three parameters (collection, key, update). The description adds no additional meaning beyond what's in the schema, such as examples or constraints on the update object. This meets the baseline for high schema coverage without extra param info.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Update') and resource ('a document in a collection'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like arango_insert or arango_remove, which also operate on documents, so it misses full sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing document key), exclusions, or comparisons to siblings like arango_insert (for creation) or arango_remove (for deletion), leaving the agent with no contextual usage cues.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arango_backupB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, indicating a safe read operation. The description adds context about creating backup files (output behavior) but doesn't detail side effects like file system changes, performance impact, or error handling. With annotations covering safety, it adds some value but lacks rich behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's appropriately sized and front-loaded, directly stating the tool's purpose without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a backup tool with readOnlyHint annotation and no output schema, the description is minimally adequate. It covers the core action but lacks details on output format, error scenarios, or integration with sibling tools. Given the complexity and annotation coverage, it's complete enough but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are fully documented in the schema. The description mentions 'collections' and 'JSON files' but doesn't add syntax, format, or usage details beyond what the schema provides. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Backup') and target ('collections to JSON files'), providing a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'arango_list_collections' or 'arango_query' which might also involve collection data retrieval, though the backup purpose is distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, timing considerations, or compare it to sibling tools like 'arango_query' for data extraction. Usage is implied by the name but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arango_create_collectionB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide destructiveHint=true, indicating this is a mutation operation. The description adds minimal context by implying a write action ('Create'), but doesn't elaborate on behavioral aspects like permissions required, idempotency, error handling, or system impact beyond what annotations cover. No contradiction with annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's front-loaded and wastes no space, making it easy to parse quickly while conveying the essential action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive creation tool with no output schema, the description is minimally adequate. It covers the basic action but lacks details on return values, error conditions, or integration with sibling tools. Given the annotations handle safety profiling, the description meets a bare minimum but doesn't provide rich contextual guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with all parameters (name, type, waitForSync) well-documented in the schema. The description adds no additional meaning about parameters, such as explaining collection naming rules, implications of 'document' vs 'edge' types, or performance effects of waitForSync. Baseline score of 3 is appropriate given the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create') and resource ('new collection in the database'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from siblings like 'arango_list_collections' (which lists rather than creates) or 'arango_insert' (which inserts data rather than creating structures), missing full sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing database access), compare to siblings like 'arango_list_collections' for checking existing collections, or specify scenarios where collection creation is appropriate versus other operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arango_insertB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide destructiveHint=true, indicating this is a write operation. The description adds minimal context by specifying 'Insert a document,' which aligns with the destructive nature but doesn't elaborate on behavioral traits like error handling, permissions needed, or idempotency. It neither contradicts nor significantly enriches the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with zero waste, front-loading the core action. It's appropriately sized for a simple tool, avoiding unnecessary elaboration while clearly stating the purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple insertion with 2 parameters), annotations cover destructive behavior, and schema fully documents inputs. However, without an output schema, the description doesn't explain return values or success/failure responses, leaving gaps in completeness for a mutation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear parameter descriptions in the schema. The description doesn't add meaning beyond the schema, such as explaining document structure constraints or collection naming rules. Baseline 3 is appropriate as the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Insert') and target ('a document into a collection'), providing a specific verb+resource combination. However, it doesn't differentiate from sibling tools like arango_update or arango_remove, which also modify collections, leaving room for improvement in distinguishing its specific role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to use this tool versus alternatives like arango_update or arango_remove, nor does it mention prerequisites such as collection existence. It lacks explicit context for usage decisions, relying solely on the tool name and basic purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arango_removeB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already declare 'destructiveHint: true', indicating this is a destructive operation. The description adds minimal value by confirming it's a removal action, but it does not disclose additional behavioral traits such as whether the removal is permanent, if it requires specific permissions, or what happens on failure (e.g., error handling). With annotations covering the destructive nature, a baseline score is appropriate, as the description provides some context but not rich behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence ('Remove a document from a collection') that efficiently conveys the core action without unnecessary words. It is front-loaded and wastes no space, making it easy for an agent to parse quickly and understand the tool's intent at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a destructive operation with 2 parameters), the annotations cover the destructive hint, but there is no output schema to explain return values. The description is minimal and does not address what the tool returns (e.g., success confirmation, error details) or other contextual aspects like rate limits or side effects. It is adequate as a basic description but lacks completeness for effective agent use without additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting both parameters ('collection' and 'key') with their types and purposes. The description does not add any semantic details beyond what the schema provides, such as examples or constraints (e.g., format of the key). Since the schema does the heavy lifting, a baseline score of 3 is justified, as the description neither compensates nor enhances parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Remove') and the resource ('a document from a collection'), making the purpose immediately understandable. However, it does not differentiate this tool from its sibling 'arango_update' (which might also modify documents) or specify what type of removal occurs (e.g., permanent deletion vs. soft delete), leaving room for improvement in sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'arango_update' for modifying documents or 'arango_query' for retrieving them. It lacks context about prerequisites (e.g., needing an existing document key) or exclusions, leaving the agent to infer usage from the tool name and parameters alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• arango_list_collectionsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint=true, which the description aligns with by implying a read operation ('List'). The description adds minimal behavioral context beyond this, such as not specifying if it returns system collections or pagination details. No contradiction exists, but with annotations covering safety, the description offers limited extra value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It front-loads the essential information ('List all collections'), making it highly efficient and easy to parse, which is ideal for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter read tool with annotations, the description is minimally adequate. However, it lacks output details (no schema provided) and doesn't address potential complexities like collection types or ordering. It meets basic needs but could be more informative given the tool's role among siblings.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately omits parameter details, focusing on the tool's purpose without redundancy. A baseline of 4 is applied since no parameters exist to explain.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('all collections in the database'), making the tool's purpose immediately understandable. However, it doesn't explicitly differentiate from siblings like 'arango_query' which might also list collections, though the verb 'List' suggests a straightforward retrieval operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, compare to siblings like 'arango_query' for more complex queries, or specify contexts where listing collections is appropriate, leaving usage decisions ambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.