Server Quality Checklist

Tool Scores

• devin_get_statusB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It does note the tool is 'non-blocking' and 'quick,' which is useful for an agent deciding whether to poll vs. block. However, it doesn't disclose rate-limit implications, whether repeated polling is discouraged, or the exact status_enum values that might be returned — the return is described but not the nuance of status values.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the one-line summary. However, the Returns section is somewhat verbose relative to value — it lists six keys that partially mirror what the output schema would already describe. The Args section is near-tautological. Some trimming could improve signal density.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter read-only tool with an output schema, the description is reasonably complete. It covers purpose, non-blocking nature, and return structure. However, it lacks guidance on how status_enum values relate to sibling tools (e.g., when to use devin_await_completion vs repeatedly calling this), and doesn't specify error conditions beyond a generic error field.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter (session_id) exists at 0% schema coverage. The description minimally states 'The session ID to check' in the args section, which is a tautology of the parameter name. No additional semantics like format, where to obtain the ID, or validation rules are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it's a 'Quick, non-blocking status check on a Devin session.' The verb 'check' with the resource 'status' is specific and clear. It distinguishes itself from siblings like devin_run_phase (execution) and devin_send_message (messaging), though it doesn't explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context via 'Quick, non-blocking' — suggesting it's for lightweight polling rather than blocking waits. However, it doesn't explicitly contrast with devin_await_completion, which would be the natural alternative for waiting on a session to finish. No explicit when/when-not guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• devin_run_phaseB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral burden. It does document the return format (success, session_id, url, error keys), which is helpful. However, it doesn't disclose side effects, auth requirements, rate limits, or what happens on partial failure. The return format disclosure is useful but incomplete for a session-spawning external tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with Args/Returns sections, making it scannable. Every sentence adds information. It's somewhat longer than the minimum, but the return-key documentation and parameter examples justify the length. Formatting is clean and front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema, which offloads some completeness burden. With only 2 parameters (one optional) and a clear purpose, the description is reasonably complete. However, given it spawns external sessions, missing context about session lifecycle, expected duration, or relationship to devin_await_completion could confuse an agent about the asynchronous nature of the created session.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema itself provides minimal help. The description compensates partially by explaining the prompt parameter with a concrete example and clarifying that playbook_id is optional if the env var is set. This adds real meaning beyond the schema, but there's no documentation of the playbook_id parameter's format or how to find valid IDs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a new Devin session with a playbook and prompt. It uses a specific verb+resource construction and the context makes the purpose distinct from siblings (get_status, await_completion, send_message). However, it doesn't explicitly differentiate itself from sibling create-like tools, though none of the siblings appear to create sessions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (starting a new session), and the example prompt provides a meaningful usage template. However, it doesn't explicitly state when to use this tool versus alternatives, nor mention exclusions or prerequisites (like needing DEVIN_PLAYBOOK_ID or an existing plan document).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• devin_send_messageA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden for behavioral disclosure. It discloses the return shape (success/detail/error) and implies the message-sending operation. However, it doesn't state whether messages require a running/awaiting session, whether there are rate limits, or what happens if the session is already complete—modest gaps. It does add a meaningful usage trigger though.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and tightly written—a purpose line, a when-to-use line, and a compact Args/Returns section. No filler words or redundancy. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite 0% schema coverage and no annotations, the description compensates well: it explains the use trigger, both parameters, and documents the return dict. An output schema exists (the Returns section), so return format needn't be elsewhere. For a simple 2-param message tool, the description is largely complete; only minor behavioral gaps (rate limits, session-state preconditions) keep it from 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and both params have minimal schema titles ('Message', 'Session Id'). The description adds modest context: session_id is 'the session ID to message' and message is 'the message content to send.' This is marginally helpful but doesn't convey format, length constraints, or expectation that the message should be a clarification prompt—acceptable but thin.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it sends a follow-up message to a Devin session with a specific verb+resource ('Send a follow-up message to a Devin session'). It distinguishes from siblings reasonably—devin_run_phase runs a phase, devin_get_status/await_completion query state—while this writes a message. It's clear but doesn't explicitly contrast with siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance: 'Use this when Devin is blocked and needs clarification to continue.' This directly answers when to use it (unblocking a stuck session). It gives clear context but doesn't mention when NOT to use it or name alternatives explicitly, which prevents a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• devin_await_completionA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the polling interval (5 seconds), the timeout default and error behavior (DevinTimeoutError), the terminal states it waits for, and the return value. It's behaviorally transparent about its blocking/polling nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly packed with useful information: polling behavior, terminal states list, timeout semantics, error behavior, return value. No filler or redundancy. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema (per context signals), so return-value documentation burden is relieved. The description covers the polling cadence, terminal states, timeout behavior, and error handling. It's fairly complete for a monitor/poll tool, though it doesn't describe what 'full session details' contains.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does address both parameters: session_id is described as 'The session ID to monitor' and timeout as 'Maximum seconds to wait' with default. This matches what's in the input schema (default 600), adding minimal extra meaning beyond restating schema values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb+resource: 'Poll a Devin session until it reaches a terminal state.' It lists the exact terminal states and distinguishes itself from siblings like devin_get_status (which presumably returns a snapshot once, not polls) and devin_run_phase. The purpose is unambiguous and specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly explains that this polls until terminal state, distinguishing it from a one-shot status check. It names the terminal states explicitly so an agent knows when to use this vs getting a single snapshot. However, it doesn't explicitly say 'use devin_get_status instead if you only need one snapshot' as an exclusionary note.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.