Server Quality Checklist

Tool Scores

• execute_codeA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses state persistence, the need for a session_id, code execution timeout limits, and the return format. However, it does not detail error handling, isolation, or what happens on invalid session.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, starting with the core behavior (persistent state) and listing parameters in a clean, readable format. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (session-based, multiple languages, timeout), the description covers essential aspects like state persistence and return values. However, it lacks guidance on choosing between execute_code and execute_terminal, and could mention error behavior more explicitly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no descriptions (0% coverage), so the description adds significant meaning: session_id is from create_session, code supports multiple languages per runtime, timeout has a maximum of 300 and default of 30. This goes beyond the schema, though language support could be more explicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it executes code in a sandbox with persistent state, similar to Jupyter cells. It mentions session_id which ties to create_session, but it does not explicitly differentiate from execute_terminal, though the context implies a session-based execution environment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for running code in a session with persistent state, and gives parameter defaults and limits. However, it does not specify when to use this tool over alternatives like execute_terminal, nor does it provide conditions when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• import_filesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears the full burden of behavioral disclosure. It describes the import process and return format but lacks details on permissions, overwrite behavior, or size limits. It is honest but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with sections and examples, though it is somewhat verbose. The return type is mentioned but redundant given the output schema. Overall, it is clear and efficiently organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and sibling tools, the description covers input formats, examples, and return value. It is mostly complete but could mention when to use this tool versus export_files or other alternatives.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description compensates well by explaining that session_id is the destination session and files is a list of objects with specific structures and examples, adding significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it imports files into the sandbox and distinguishes between host path and cross-session imports, specifying the required fields for each. This provides a specific verb and resource, differentiating from siblings like export_files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use each entry format (host path vs cross-session) and notes that cross-session files must have been previously exported. However, it does not explicitly state when not to use this tool or compare directly with sibling tools like create_session or export_files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_sessionA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses isolation (Docker) and emptiness, but omits details like rate limits, authentication needs, or whether existing sessions are affected. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Compact and well-structured: one-line purpose, followed by usage tip, then formatted Args and Returns sections. Every sentence is informative with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, parameter usage, and return format. Missing details like error handling or resource limits, but given low complexity (2 params, output schema present) and sibling tools, it is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, but description adds complete semantics: language options (python, node, r), dependency structure (package names, version strings, use quotes, null handling), and example. Adds high value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it creates an isolated sandbox environment (Docker container), with a specific verb and resource. Distinguishes from sibling tools (execute, file ops, stop) which focus on other actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: sandbox starts empty so specify all needed packages, and for data analysis include pandas and numpy. Implies when to use, but does not explicitly state when not to use or list alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• stop_sessionA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It indicates complete removal, but does not mention irreversibility or potential data loss explicitly, which would enhance transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely concise with three sentences covering purpose, usage, parameters, and returns. No wasted words, front-loaded with the main action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and an output schema, the description is complete: it explains the action, when to use, the parameter source, and return format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds essential meaning by stating session_id is the ID returned by create_session, compensating for the schema's 0% coverage and clearly indicating the parameter's source.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool stops and removes the sandbox completely, which distinguishes it from sibling tools like create_session or execute_code.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises using it when done with a session, providing clear usage context. Does not mention when not to use or alternatives, but this is sufficient given the simple action.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_filesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses key behaviors: no host copy, API must be running, and per-file result details. Lacks auth or rate limit info.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise with clear summary, args, returns. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, args, and returns adequately. Could mention error handling but overall complete given moderate complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description explains session_id and files with an example, adding significant meaning beyond raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool registers files for download and cross-session import, distinguishing it from siblings like import_files.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains that files become available via download endpoint and for import_files in other sessions, but does not explicitly state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_terminalA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that commands run inside a sandbox, requires a session ID from create_session, and returns stdout, stderr, and exit_code. It hints at mutability (installing dependencies) but could explicitly mention that the sandbox state persists across commands.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear purpose statement, usage context, parameter explanations, and return format. Every sentence adds value and is well-structured without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the two simple parameters and presence of an output schema, the description fully covers required context: it explains the sandbox environment, session prerequisite, typical use cases, and return structure. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description fully compensates by explaining that session_id is the ID from create_session and command is a shell command with examples. This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a shell command inside a sandbox, with examples like listing files and installing dependencies. It distinguishes itself from sibling tools like execute_code by focusing on shell commands, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides specific use cases (listing files, installing dependencies) and implies when to use it. However, it does not explicitly state when not to use it or mention alternative tools like execute_code for non-shell scripts, leaving some room for ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.