Server Quality Checklist

Tool Scores

• count_tokens_fileA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the burden of behavioral disclosure. It does add useful details: the file must be UTF-8, and an empty model defaults to ANTHROPIC_TOKEN_COUNT_MODEL. It does not describe what happens with invalid paths, encoding errors, or file-size limits, but for a straightforward read-only counting operation this is acceptable. The description adds some behavioral context without being exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tight and front-loaded: a one-line purpose followed by an Args section. There is no filler, and each sentence adds value. It covers both parameters and the default behavior without unnecessary elaboration, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter tool, the description covers the essential inputs and default behavior. The output schema exists, so return-value documentation is not required from the description. However, it lacks guidance on error conditions, what constitutes a valid UTF-8 file, and how to choose between this and sibling counting tools, leaving moderate gaps for an agent that lacks prior context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does meaningfully. It clarifies file_path must be an absolute path, and it explains model as 'Model or encoding name' with an explicit default behavior. Both parameters are given semantic meaning beyond their names, types, and defaults, making the tool usable without inspecting external documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Count tokens') on a specific resource ('a UTF-8 file'), which is immediately distinguishable from sibling tools like count_tokens_folder, count_lines_file, count_chars_file, and count_kb_file. The file path parameter and encoding qualifier make the scope explicit. Unlike a vague 'Process' definition, this gives an agent a clear understanding of what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool should be used when a token count for a file is needed, and the sibling list provides alternative counting tools. However, there is no explicit guidance on when to prefer this over count_tokens or count_tokens_folder, nor any exclusions for non-UTF-8 files or other edge cases. Usage context is present but left largely to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_chars_fileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It provides useful behavioral detail by specifying UTF-8 encoding and the Unicode code point counting semantics, but it does not mention error behavior for invalid UTF-8, missing files, or whether newlines are included. This is adequate for a simple read operation but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two lines long, front-loads the core behavior, and contains no filler. The 'Args' clarification about absolute paths earns its place by guiding correct invocation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter counting tool with an output schema available, the description covers the necessary input semantics and the key counting behavior. It does not describe edge cases or error handling, but those are relatively minor for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does by documenting file_path as an absolute path, which adds meaningful constraint beyond the schema's bare 'File Path' string type. This is sufficient for the tool's single parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Count') and resource ('UTF-8 file'), and clarifies that it counts Unicode code points rather than bytes or lines. This clearly differentiates it from sibling tools like count_tokens_file, count_lines_file, and count_kb_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose implies when to use this tool — when a Unicode code point count of a file is needed — but it does not explicitly mention alternatives or state conditions for choosing another sibling tool. Usage guidance is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_lines_fileA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the behavioral disclosure burden. It usefully discloses splitlines semantics, including the trailing-newline edge case, and requires an absolute path. It does not mention error or encoding behavior, but for a simple read-only counting tool the stated behavior is meaningful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, front-loads the critical semantic rule, and includes only a single necessary parameter explanation. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, an existing output schema, and the disclosure of the important splitlines behavior, an agent has enough information to call it correctly. Minor gaps remain around when to choose it over sibling tools and how errors are reported, but these are not blocking for this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no description coverage, but the description's args section defines file_path as "Absolute path to the file," adding a real constraint beyond the schema's bare string type. This adequately compensates for the schema gap for a single-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  "Count lines in a file" states a specific verb, resource, and metric. The added splitlines semantics further disambiguates exactly what is counted, and the tool name and siblings make the distinction from count_tokens_file and count_chars_file clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance on when to select this tool over sibling tools such as count_tokens_file or count_chars_file. Usage is only implied by the name and purpose, with no mention of alternatives or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_kb_fileA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It adds meaningful behavioral detail by specifying the rounding method and the Windows Explorer compatibility, which goes beyond simply saying 'returns file size'. It does not disclose error behavior or what happens for non-file paths, but the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: one sentence defining behavior and formula, followed by the single argument's purpose. Every word adds value, and the key behavioral detail is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter tool with an output schema, the description provides the essential formula, the matching behavior, and the path requirement. It is nearly complete, but omits edge-case behavior such as nonexistent files, directories, or permission failures.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It does so by specifying that file_path is an 'Absolute path to the file', adding a meaningful constraint beyond the schema's bare string type. Since there is only one parameter and its role is clear, this is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool computes file size in KB using a specific formula, ceil(bytes / 1024), and explicitly matches Windows Explorer's display convention. This distinguishes it from sibling tools that count tokens, lines, or characters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage context is implied by the name and description: use this when you need a file's size in KB. However, there is no explicit guidance about when to prefer this tool over siblings, no exclusions, and no mention of invalid inputs such as nonexistent files or directories.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_tokensA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden. It discloses two meaningful behaviors not visible in the schema: tokenizer selection is routed by model name with an Anthropic default, and an empty model falls back to ANTHROPIC_TOKEN_COUNT_MODEL. This is useful context beyond a bare 'count tokens' statement, though it does not mention failure modes for unsupported model names.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences plus a compact Args list; every line adds information (purpose, routing, parameter behavior). The core purpose is front-loaded and no filler is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple token-counting tool with an output schema present, the description covers purpose, parameter semantics, and routing. It does not explicitly address edge cases (e.g., unknown model, empty text), but the complexity is low enough that these omissions are not harmful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must clarify the parameters, and it does. 'text' is explained only minimally, but 'model' is well specified with concrete examples (claude-opus-4-8, gpt-4, o200k_base) and the empty-string fallback behavior, adding real semantic value over the schema's bare type/default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('Count tokens in text'), which directly differentiates it from the file/folder siblings by scoping to raw text. The model-routing clause adds useful specialization without making the purpose ambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use case through 'in text' and the sibling names suggest file/folder alternatives, but it never explicitly states when to choose this tool over count_tokens_file/count_tokens_folder or mentions any exclusion criteria. The routing and default model info is model-selection guidance, not tool-selection guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_tokens_folderA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations to lean on, the description discloses key behaviors: skipped dependency/build directories, binary file detection strategy, encoding fallbacks, and the aggregated return value. This is unusually thorough and goes well beyond a simple statement of the operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-organized: a one-sentence summary, followed by behavioral details, then an Args section. Every sentence carries relevant information with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool, the description fully covers inputs, exclusions, encoding handling, and return semantics. Even with an output schema present, the explicit statement that it returns the sum of per-file counts removes any ambiguity about the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must do all the work. It explains folder_path must be an absolute path and clarifies the model parameter, including its default env-var behavior. This adds meaningful semantic value beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence names the specific operation: counting tokens across all text files in a folder, recursively. This clearly distinguishes it from sibling tools like count_tokens_file or count_lines_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The recursive folder-wide scope strongly implies when to use this tool versus single-file alternatives, but it never explicitly names the alternatives or states when not to use it. The usage context is clear from the description, though exclusions are absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.