Server Quality Checklist

Tool Scores

• fsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses important behavioral traits: 'UNSANDBOXED, 50MB read limit' and 'Auto-fixes code fences and smart quotes on write.' However, with 31 parameters and many actions, it lacks detail on side effects or specific behaviors for each action. Given no annotations, the description carries the burden but provides moderate transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but front-loaded with critical info (unsandboxed, read limit). It then lists all actions without clear structure or grouping. While efficient, it could be better organized for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the full range of 19 actions, important constraints (read limit, no race conditions), and additional features (auto-fix). With no output schema, it provides sufficient context for the tool's capabilities, though more detail on return values for each action would enhance completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 84%, the schema already documents most parameters. The description adds minimal value, e.g., listing actions and mentioning 'edit: [{old_text, new_text}]' but does not enhance understanding beyond the schema. It misses opportunities to clarify parameter usage or dependencies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Full shell filesystem access (UNSANDBOXED, 50MB read limit). Use for all file operations.' The tool's purpose is specific and distinct from sibling tools js_exec, python_exec, and terminal, which are code execution environments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises 'Use for all file operations' and notes that it 'Doesn't encounter race conditions with file watchers.' It also mentions that 'grep contains all the functionality of bash + powershell filesystem search.' This provides good context for when to use this tool, although it does not explicitly exclude scenarios or mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• terminalA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses return format, unrestricted privileges, no sandboxing, shell differences (Windows cmd.exe, Unix bash), timeouts (120s default, 600s max), and background task management. Could add explicit warning about destructive potential, but overall strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise yet comprehensive; each sentence adds unique information. Front-loaded with core purpose, then progressively details. No redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 8 parameters, no output schema, and no annotations, the description covers purpose, usage, behavior, return format, timeouts, background tasks, cross-platform, and alternatives. Highly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (baseline 3). Description adds value by grouping background-related params (bg_status, bg_kill, bg_list) and clarifying timeout defaults and max, beyond schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute shell commands' and enumerates specific use cases (git, npm, pip). It distinguishes from siblings by advising 'For file ops prefer fs; for Python prefer python_exec.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly lists when to use ('Use for: git, npm, pip, system commands, anything requiring shell') and provides exclusions ('For file ops prefer fs; for Python prefer python_exec'). Also warns of unrestricted nature.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• js_execA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: return object structure, persistent state mechanism, timeouts, source/line limits, output cap, error handling, and requirement for Node.js on PATH. This is comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections (purpose, output, usage, state, limitations), but slightly verbose. Could tighten some examples, but overall effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and two parameters, the description covers all critical aspects: purpose, usage, output, state handling, and limitations, leaving no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds value by explaining state persistence and providing an example for the 'code' parameter, going beyond what the schema descriptions offer.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that it 'Execute JavaScript code in a full Node.js environment' and provides four specific use cases (JSON manipulation, algorithms, npm packages, Node.js APIs) that distinguish it from siblings like python_exec and fs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells when to use (four bullet points) and when NOT to use (two bullet points, e.g., visualizations and Python-preferred tasks), giving clear guidance on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• python_execA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully covers behavior: full system access, allowed imports, persistent state, limits (timeout, chars, lines, AST nodes). It discloses permissions and constraints comprehensively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with no wasted words. It is front-loaded with purpose, then usage, then details, using bullet points for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema or annotations, the description is complete: it covers purpose, usage, behavioral traits, parameter details, limits, and expected output format. It leaves no significant gaps for a code execution tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds valuable semantics for the 'code' parameter (use print() for output, last expression returned in 'result'), going beyond the schema. 'extended_imports' is adequately described.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute Python code with full system access,' specifying the verb and resource. It lists example use cases (data processing, file operations, etc.) and distinguishes from js_exec, making the purpose highly specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use for: data processing... any Python task' and 'Prefer over js_exec unless JS-specific features needed,' providing clear guidance on when to use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.