Server Quality Checklist

Tool Scores

• matrix_get_room_infoB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true, covering safety and idempotency. The description adds context about what information is returned (room name, topic, creation info, member list), which is useful but doesn't disclose behavioral traits like rate limits, authentication needs, or error conditions beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by return details and parameter info in a structured format. It's efficient with minimal waste, though the Args and Returns sections slightly duplicate schema information without adding value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema) and rich annotations, the description is adequate but incomplete. It lacks usage guidelines and doesn't fully compensate for the absence of an output schema by detailing return value structures or error handling, leaving gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'room_id' fully documented in the schema as 'The Matrix room ID'. The description repeats this information in the Args section without adding meaningful semantics beyond the schema, such as format examples or validation rules.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and resource 'detailed information about a specific Matrix room', specifying it returns room name, topic, members, and other state information. However, it doesn't explicitly differentiate from sibling tools like 'matrix_list_rooms' (which likely lists multiple rooms) or 'matrix_get_messages' (which focuses on messages rather than room metadata).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'matrix_list_rooms' for listing rooms or 'matrix_get_messages' for message retrieval, nor does it specify prerequisites or contextual constraints for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• matrix_search_messagesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, open-world, idempotent, and non-destructive behavior, so the description does not need to repeat these. It adds valuable context by specifying the search scope ('all rooms or a specific room') and return format ('Matching messages with room, sender, and content'), enhancing understanding beyond annotations without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by structured 'Args' and 'Returns' sections. Every sentence is necessary and efficient, with no redundant information, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, rich annotations, and 100% schema coverage, the description is largely complete. It explains the purpose, usage, parameters, and return values. However, without an output schema, it could benefit from more detail on the structure of returned messages, though the annotations provide safety context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter well-documented in the schema. The description adds minimal value by summarizing parameters in the 'Args' section but does not provide additional semantics, syntax, or format details beyond what the schema already covers, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search for messages'), resource ('across Matrix rooms'), and scope ('all rooms or a specific room'), distinguishing it from sibling tools like matrix_get_messages (which likely retrieves messages without search) and matrix_list_rooms (which lists rooms rather than searching within them).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for usage ('Search for specific content within messages') and implies when to use it (for searching rather than direct retrieval), but it does not explicitly name alternatives or state when not to use it, such as distinguishing from matrix_get_messages for non-search scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• matrix_list_roomsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true, covering safety and idempotency. The description adds valuable context beyond annotations: it specifies the return format (room IDs, names, topic, member count) and mentions the purpose of discovering rooms before fetching messages, which helps the agent understand the tool's role in a workflow. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by return details and usage guidance. Every sentence adds value: the first states what it does, the second specifies returns, and the third provides context for usage. No wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one optional parameter), rich annotations (covering safety and idempotency), and no output schema, the description is mostly complete. It explains the purpose, returns, and usage context. A minor gap is lack of explicit mention of pagination or ordering for the list, but annotations (openWorldHint=true) imply scalability, and the limit parameter handles basic control.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter 'limit' fully documented in the schema (type, range, default). The description does not add any parameter-specific information beyond what the schema provides, so it meets the baseline of 3 for high schema coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all Matrix rooms') and resource ('that the user has joined'), distinguishing it from siblings like matrix_get_room_info (which likely gets details for a specific room) and matrix_search_messages (which searches messages rather than listing rooms). It explicitly mentions the verb 'list' and scope 'joined rooms'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('to discover which rooms are available before fetching messages') and names a specific alternative ('Use room_id with matrix_get_messages to read messages'). It also implies when not to use it (e.g., for room details or message searching, handled by other siblings).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• matrix_get_messagesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, openWorldHint=true, and idempotentHint=true, covering safety and idempotency. The description adds valuable behavioral context: pagination behavior (tokens, direction options), return format details (list with sender/timestamp/content), and default values (limit default 50), which aren't covered by annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded: the first sentence states the core purpose, followed by usage guidance, then parameter explanations, and return details. Every sentence earns its place with no wasted words, making it efficient and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, pagination), rich annotations (covering safety/idempotency), and 100% schema coverage, the description is complete. It adds necessary context like prerequisite usage, behavioral details (pagination, returns), and distinguishes from siblings, compensating for the lack of output schema by describing return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all parameters. The description adds minimal extra semantics: it clarifies 'from' as a pagination token from previous requests and mentions default values, but doesn't provide significant additional meaning beyond the schema's thorough descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'retrieve' and resource 'messages from a specific Matrix room', distinguishing it from siblings like matrix_get_room_info (room metadata), matrix_list_rooms (room listing), and matrix_search_messages (filtered search). It specifies the exact scope of retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides when-to-use guidance: 'Use matrix_list_rooms first to get available room IDs.' This directs users to a prerequisite tool and distinguishes it from matrix_search_messages by implying this is for bulk retrieval rather than filtered search.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.