Server Quality Checklist

Tool Scores

• run_filterA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: it loads a file, converts to in-memory document, calls filter_item, and returns text. It details file type handling, loading behavior per type, errors (ValueError, FileNotFoundError), and the return structure. This is comprehensive and transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, usage, supported types, behavior, args, returns, raises). Every sentence adds value, though it is slightly verbose. It is front-loaded with purpose, making it easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (not shown but noted), the description appropriately does not repeat return values. It covers prerequisites (register_filter), file types, loading behavior, errors, and the output structure. It is fully complete for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by explaining the file loading behavior per type (e.g., 'json -> parsed JSON value'), which enriches understanding of the file_type and file_path parameters beyond the schema. This extra context justifies a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run a previously registered filter on a local file and return its text output.' It specifies the action (run), the resource (filter), and the context (on a local file). It also distinguishes from sibling tools by mentioning its dependency on register_filter and the distinct behavior from convert_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this tool after register_filter,' guiding the agent on the correct sequence. It implies the prerequisite without explicitly excluding alternatives, but the context is clear. It does not mention when not to use or compare to convert_file, but the guideline is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• convert_fileA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It thoroughly discloses all behavioral traits: writing behavior, workdir requirement, automatic directory creation, overwrite rejection (unless overwrite is true), and specific error conditions (ValueError, FileNotFoundError). No contradictory information.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a clear one-sentence summary, followed by usage context, then parameter explanations, and finally returns/raises. It is structured and relatively concise, though a bit lengthy. Every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, 3 required), high schema coverage (100%), and presence of output schema, the description is complete. It covers purpose, usage, parameter details, return values, and error conditions. No gaps for the agent to make incorrect decisions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value beyond schema: explains that file_type is an optional override, source_file_path must be inside allowed workdir, missing parent directories for destination are created automatically, and overwrite defaults to false. This enhances the agent's understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Apply a registered filter to a source file and save the text output.' It specifies the resources (source file, destination file, filter) and explicitly contrasts with sibling tools by indicating it is used after register_filter and that run_filter is the alternative for non-persistent transformation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance: use after register_filter, for transforming local json/yaml/txt files and persisting output. It also specifies when not to use: 'refuses to write when no --workdir is configured.' It mentions automatic parent directory creation and overwrite behavior, giving clear context for decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• register_filterA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits: validation, restricted Python subset, preloaded modules, safety rules, expiration via TTL, and forbidden actions. This is comprehensive beyond what annotations typically provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with sections but somewhat lengthy for a tool with one parameter. However, each section adds necessary clarity for a complex tool, so it earns a 4 for being appropriately detailed without bloat.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (Python code validation, constraints, multiple input types), the description covers all essential aspects: usage order, function signature, input handling, safety rules, and return format. Complete even with output schema present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (single param 'code'), but description adds rich semantics: explains the code must define filter_item, specifies function signature, input types, and required return type. This significantly enhances the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it validates and registers a restricted Python filter for later execution on a local file. It distinguishes from siblings by explicitly saying 'Use this tool first' and contrasts with run_filter.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Use this tool first when you want to run custom filtering or transformation logic against a local document.' It implies when not to use (for execution, use run_filter) and gives alternative context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.