Server Quality Checklist

Tool Scores

• list_tablesA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It usefully specifies the return format (JSON string with table names and types), but omits safety characteristics (read-only), rate limits, or error conditions that annotations would typically cover.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely concise with two distinct, information-dense segments: purpose declaration and return value specification. No redundant or filler text; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (zero parameters) and existence of an output schema, the description adequately covers the essential contract. However, the absence of usage guidance and behavioral annotations leaves minor gaps in contextual completeness for an agent deciding between discovery tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Zero parameters present, which per calibration establishes a baseline of 4. The description correctly omits parameter discussion since none exist, and the schema coverage is trivially complete at 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (List) and resource (tables/indexes in Manticore Search), providing specific context. However, it does not explicitly differentiate from sibling 'describe_table' (list vs. describe single table), though the distinction is implied by the verb choice.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance provided on when to use this versus 'describe_table' or 'run_query'. For example, it does not indicate whether this should be called before querying or how it relates to schema discovery workflows.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_tableA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the return structure ('A dictionary containing column information'), but fails to explicitly state safety properties (read-only/non-destructive), error conditions (e.g., table not found), or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description uses a standard docstring format (Args/Returns) which is efficiently organized. The main purpose is front-loaded in the first sentence. While the Args/Returns format duplicates schema concepts, it is necessary here due to the zero-coverage schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter introspection tool, the description adequately covers the input parameter and return value structure. It appropriately compensates for the lack of schema descriptions. Minor gap: no mention of error handling for invalid table names.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, leaving the parameter completely undocumented in structured data. The description compensates effectively by defining table_name as 'The name of the table/index to describe', providing necessary semantic context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise verb ('Get') and resource ('schema of a specific table/index'), and specifies the system context ('Manticore Search'). This clearly distinguishes it from siblings: list_tables (lists tables), run_query (executes queries), and documentation tools (retrieves docs).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites (e.g., using list_tables first to discover valid names) or when to prefer this over run_query for schema introspection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_queryA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full disclosure burden. It effectively communicates the default read-only safety behavior, the mechanism for enabling write access (environment variable), and documents the return structure (dictionary with columns/rows/total). It lacks mention of error handling or rate limits, preventing a perfect score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with clear docstring-style sections (Args, Returns). The purpose is front-loaded in the first sentence, followed by critical behavioral constraints, then parameter details. Every sentence provides necessary information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's single-parameter simplicity but behavioral complexity (read-only vs. write modes), the description is complete. It covers invocation requirements, parameter semantics, execution behavior, and return values, providing sufficient context for correct agent operation despite the lack of annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, requiring the description to fully compensate. It successfully adds semantic meaning by documenting the 'query' parameter as 'The SQL query to execute' and provides a concrete example ('SELECT * FROM my_index LIMIT 10'), effectively bridging the schema documentation gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Execute a SQL query against Manticore Search,' providing a specific verb (Execute), resource (SQL query), and target system (Manticore Search). This clearly distinguishes it from sibling tools like describe_table and list_tables, which handle metadata rather than execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage constraints regarding read-only mode by default and the MANTICORE_ALLOW_WRITE_ACCESS environment variable requirement for DDL/DML. However, it does not explicitly differentiate when to use this tool versus sibling introspection tools like describe_table, though the behavioral guidance for write operations is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_documentationA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Adds valuable behavioral context: 'file list is cached, so filtering is instant and local' and describes return format ('grouped by category'). Does not explicitly state read-only/safety status, though implied by operation type.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Lengthy but high information density. Structure is logical: purpose → critical usage warning → technical note → parameter details → examples. Every sentence earns its place, including the IMPORTANT call-efficiency warning which prevents API misuse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Comprehensive for tool complexity (1 param, output schema exists). Covers purpose, parameter semantics (since schema lacks them), behavioral traits (caching), and return structure. Output schema presence excuses minimal return value description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (parameter has no description), but description extensively compensates: explains regex pattern support, case-insensitivity, OR syntax with pipe (|), directory prefix matching (^), and provides 6 concrete examples covering multiple use cases.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb ('List') + specific resource ('documentation files from Manticore Search manual'). Distinguishes from sibling 'get_documentation' (likely retrieves content) and 'list_tables' (different domain).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Excellent explicit guidance: 'Make ONE call with regex OR patterns' vs 'DO NOT make multiple calls', with concrete EFFICIENT/INEFFICIENT examples. Clearly states when to use pipe patterns versus separate calls.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_documentationA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It successfully discloses the return format ('markdown text'), the filtering behavior ('returns only matching sections'), and the context line logic for the before/after parameters. It does not explicitly declare read-only/safety properties or error handling, but 'Fetch' implies a safe read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Appropriately structured with clear sections (implied Args, Returns, Examples) using Google-style docstring formatting. Every sentence earns its place: the opening defines purpose, the Args section explains parameters, Returns clarifies output format, and Examples demonstrate concrete usage. No waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters with 0% schema coverage and an output schema existing, the description is complete. It documents all parameters, explains the return type (leaving detailed schema to the output_schema), provides usage examples, and establishes ecosystem context with sibling tools. No gaps remain for this complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, requiring the description to fully compensate. It excellently documents all 4 parameters: file_path includes a concrete example ('Searching/KNN.md'), content explains the filtering semantic, and before/after clarify they represent line counts with default values. This fully compensates for the schema deficiency.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb and resource ('Fetch documentation from Manticore Search manual'). It effectively distinguishes from sibling tools by explicitly referencing list_documentation() as a prerequisite discovery mechanism, establishing the workflow relationship between the two.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use this tool versus its sibling: 'Use list_documentation() first to discover available files.' This provides clear prerequisite guidance. The examples further clarify usage patterns showing both filtered and unfiltered invocation patterns.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.