Server Quality Checklist

Tool Scores

• list_devin_sessionsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure but offers minimal information. It doesn't mention whether this is a read-only operation, what permissions might be required, whether results are paginated (though parameters suggest they might be), or what format the returned sessions will have. The description adds almost no behavioral context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise - a single sentence with no wasted words. It's front-loaded with the essential information (the action and resource) and contains no unnecessary elaboration. This is an example of efficient communication where every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there are no annotations and no output schema, the description is incomplete for a tool that presumably returns session data. For a list operation with pagination parameters, the description should ideally mention what information sessions contain, whether results are ordered, or what authentication might be required. The minimal description leaves significant gaps in understanding the tool's behavior and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already fully documents both parameters (limit and offset). The description adds no additional parameter semantics beyond what's in the schema, so it meets the baseline of 3 for high schema coverage situations where the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all Devin sessions'), making the purpose immediately understandable. However, it doesn't differentiate this tool from potential siblings like 'get_devin_session' or explain what distinguishes 'list' from 'get' operations in this context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'get_devin_session' (presumably for retrieving a specific session) and 'send_message_to_session' (for interacting with sessions), the description offers no context about when listing all sessions is appropriate versus using more targeted tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_organization_infoB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is a 'Get' operation, implying it's likely read-only, but doesn't confirm if it requires authentication, has rate limits, or what specific information is returned. For a tool with zero annotation coverage, this is inadequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It's front-loaded with the core purpose and appropriately sized for a simple tool, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is minimally adequate. It states what the tool does but lacks details on return values, authentication, or error handling, which could be helpful for an agent. It meets the bare minimum but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100% (as there are no parameters to describe). The description doesn't need to add parameter semantics, so a baseline score of 4 is appropriate, as it avoids redundancy while being complete for this case.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('information about the current Devin organization'), making the purpose unambiguous. However, it doesn't differentiate from sibling tools like 'get_devin_session' or 'list_devin_sessions' that might also retrieve organizational data, so it doesn't reach the highest score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, context, or exclusions, leaving the agent to infer usage from the tool name alone. This is a significant gap in helping the agent select the right tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_devin_sessionB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves information, which implies read-only behavior, but fails to disclose critical traits such as authentication requirements, rate limits, error conditions, or what 'information' includes beyond Slack messages, leaving significant gaps for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core purpose ('Get information about an existing Devin session') and efficiently adds the optional feature ('and optionally fetch associated Slack messages'), with zero wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete for a tool with 2 parameters and behavioral complexity. It doesn't explain what 'information' is returned, how errors are handled, or prerequisites like session existence, leaving the agent with insufficient context for reliable invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with clear descriptions for both parameters (session_id and fetch_slack_info). The description adds minimal value beyond the schema by mentioning 'optionally fetch associated Slack messages', which aligns with fetch_slack_info, but doesn't provide additional context like format or constraints, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Get information about an existing Devin session') and resources ('Devin session', 'associated Slack messages'), distinguishing it from siblings like create_devin_session (creates new), list_devin_sessions (lists multiple), and send_message_to_session (sends messages).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by mentioning 'existing Devin session' (suggesting it's for retrieval rather than creation) and 'optionally fetch associated Slack messages' (hinting at an alternative use case), but it lacks explicit guidance on when to use this versus alternatives like list_devin_sessions or detailed exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• send_message_to_sessionA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the action 'send a message' but doesn't describe what happens after sending (e.g., does Devin respond, is it async, are there rate limits, what permissions are needed). For a communication tool with zero annotation coverage, this leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose and includes the optional Slack functionality. Every word earns its place with no redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description adequately covers the basic action but lacks details on behavioral outcomes, error conditions, or response format. For a tool that sends messages (potentially involving async interactions or notifications), more context on what to expect after invocation would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 4 parameters thoroughly. The description adds no additional meaning beyond what's in the schema (e.g., it doesn't explain the relationship between slack_channel and slack_thread_ts or provide usage examples). Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'send' and the resource 'message to an existing Devin session', with additional optional functionality 'to the associated Slack thread'. It distinguishes from siblings like create_devin_session (creation vs. communication) and get/list sessions (retrieval vs. action).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have an existing session (vs. create_devin_session for new sessions) and want to communicate with it, but doesn't explicitly state when to use this tool versus alternatives like whether it's for real-time interaction or notifications. No explicit exclusions or detailed context are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_devin_sessionA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that the tool posts to Slack automatically and requires language consistency, which are valuable behavioral traits. However, it doesn't cover important aspects like whether this is a mutating operation, what permissions are needed, rate limits, or what happens on failure, leaving gaps for a creation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with two sentences that each serve distinct purposes: the first states the core functionality, the second provides implementation guidance. It's front-loaded with the main action and avoids unnecessary repetition, though the second sentence could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters, no annotations, and no output schema, the description provides adequate context about the creation and Slack posting behavior. However, it lacks information about what the tool returns (session ID, status, etc.), error conditions, or dependencies, making it incomplete for a creation tool with multiple parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 5 parameters thoroughly. The description doesn't add any parameter-specific information beyond what's in the schema, such as explaining how 'prompt' should be structured or when to use 'machine_snapshot_id'. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a new Devin session for code development and posts the task to Slack, providing a specific verb ('create') and resource ('Devin session'). It distinguishes from siblings like 'get_devin_session' or 'list_devin_sessions' by focusing on creation rather than retrieval, though it doesn't explicitly contrast with 'send_message_to_session'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool: it's the 'recommended approach' for creating sessions that automatically post to Slack as @Devin mentions. It includes guidance on language consistency, but doesn't explicitly state when not to use it or mention alternatives like using other session tools without Slack integration.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.