Server Quality Checklist

Tool Scores

• search_findingsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It only lists return fields and an output instruction (URL inclusion), but omits details on side effects, permissions, rate limits, error behavior, or the fact that it's read-only. The description does not confirm safety or non-destructiveness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences plus a brief note. It front-loads the primary action and return values, and the important instruction about URLs is clearly separated. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (12 parameters, nested object, no output schema) and no annotations, the description is somewhat incomplete. It explains what the tool does and returns, but lacks guidance on parameter usage, pagination behavior, or how to construct effective queries. The return field list is helpful, but users may need more context for a tool with this many filters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds minimal value beyond the schema: it lists return fields but does not elaborate on parameter meaning, usage patterns, or how to combine filters effectively. The instruction about the URL relates to output, not parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool searches 'Solodit's 20k+ smart contract security findings' and lists return fields (severity, firm, etc.). It clearly differentiates from siblings like get_finding (single finding) and get_filter_options (filter options) by focusing on search over multiple audits.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use (searching findings) and an important presentation guideline ('always include the Solodit URL'). However, it does not explicitly contrast with alternatives (e.g., when to use get_finding instead), leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_findingA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears the full burden. It states what it returns but does not disclose behavioral traits such as error handling (e.g., what if identifier is invalid), rate limits, authentication requirements, or side effects. This is a significant gap for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, concise and front-loaded. Every sentence is essential, no filler. Efficiently communicates purpose and key details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description provides a reasonable level of completeness for a fetch tool. It explains what it does, the input format, and what it returns. However, it lacks details on error behavior and prerequisites, which would be helpful for robust usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers the parameter fully with a description of acceptable identifier types. The description adds value by recommending numeric ID from search results, which aids correct invocation. Schema coverage is 100%, baseline 3, and the added guidance justifies a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves full details for a specific Solodit finding, specifying three types of identifiers (numeric ID, URL, slug) and that it returns complete markdown and metadata. This distinguishes it from sibling tools like search_findings which is for searching, not fetching full details.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for obtaining full details of a specific finding and recommends preferring numeric ID from search results, but does not explicitly state when not to use it or provide alternatives. It lacks explicit when-to-use and when-not-to-use context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_claudit_versionA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It accurately states the tool returns the version, implying a safe read operation with no side effects. No additional context like auth requirements or rate limits is given, but the simplicity of the tool makes this acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the key information. No superfluous words, every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has zero parameters and no output schema, the description fully covers what the tool does. It leaves no ambiguity for the agent about the purpose and result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema description coverage is 100%. According to guidelines, 0 parameters baseline is 4. The description adds no parameter information, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns the currently running version of the Claudit MCP server, using a specific verb ('Returns') and resource ('currently running version'). It clearly distinguishes from siblings like 'get_filter_options' and 'search_findings' which deal with filtering/searching, not version info.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when-to-use or when-not-to guidance is provided, but for a simple version retrieval tool the context is clear. The description implies it should be used when the agent needs to know the server version, and no alternatives are mentioned. Lacks explicit exclusions but is adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_filter_optionsA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It states the tool lists values but does not describe output format, performance characteristics, or how the list is generated (static vs dynamic). For a simple list operation, the lack of detail is acceptable but not exemplary.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no redundancy. The first sentence states purpose, and the second provides usage context. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no output schema), the description is mostly complete. It could specify the output structure (e.g., whether it returns a list of strings or a mapping), but the current level is sufficient for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the description adds meaning beyond the empty schema by explaining what the output represents (filter categories) and how it relates to search_findings. This exceeds the baseline of 4 for zero-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists available filter values for Solodit search, specifying the categories (firms, tags, categories, languages). It distinguishes from sibling tool 'search_findings' by explicitly linking the output to its filters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to discover valid values for search_findings filters,' providing clear guidance on when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.