Server Quality Checklist

Tool Scores

• mat_parse_reportC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It states 'run headlessly' but does not disclose side effects, authentication needs, resource consumption, or what 'artifacts' are returned. Minimal behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no redundancy. However, it is under-specified; conciseness is not valued over informativeness. Adequate but not optimal.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool has 5 parameters including nested objects, no output schema, and no annotations. Description fails to explain return value format, parameter meanings, or preconditions. Severely incomplete for this complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and description provides no parameter information. The five parameters (heap_path, report_id, xmx_mb, options, timeout_sec) are completely unexplained. Baseline for low coverage requires compensation, which is absent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses specific verb 'Run' and resource 'predefined MAT report', mentions 'headlessly' and 'return generated artifacts'. It distinguishes from sibling tools like mat_oql_query and mat_healthcheck, but could be more explicit about what a MAT report is.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives. Does not specify prerequisites, context, or when not to use. Agent must infer usage from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mat_oql_queryC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full burden of disclosing behavioral traits. It only states that it executes a query and returns results, but does not mention potential side effects, auth requirements, rate limits, or error handling expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short and front-loaded with the key action. It is concise and contains no filler, though it could be slightly more informative without losing brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's 7 parameters, no output schema, and no annotations, the description is insufficient. It does not explain the purpose of OQL, the role of heap_path, or what 'result artifacts' entail, leaving critical gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds no meaning beyond the parameter names. The single sentence does not explain the purpose or effect of any of the 7 parameters (e.g., heap_path, oql, limit, format).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool executes a single MAT OQL query and returns result artifacts. It uses a specific verb and resource, and the sibling tools suggest different purposes (e.g., healthcheck, spec), but the description does not explicitly distinguish itself from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like mat_oql_spec or mat_healthcheck. There is no mention of prerequisites, when not to use it, or recommended use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mat_healthcheckC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It states the tool validates, but it does not explain side effects (likely none), required permissions, rate limits, or output format. With zero behavioral context, the AI agent cannot anticipate the tool's impact or constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that communicates the core purpose without extraneous words. While it is very brief, it earns its place by being direct. However, it could be slightly more informative without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of output schema and annotations, the description should provide a more complete picture of what the tool returns (e.g., success/failure status, details of validation). It also omits any information about error handling or the scope of validation. The sibling tools suggest a suite of MAT-related operations, but this description does not integrate the health check into that context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the input schema provides no parameter descriptions. The tool description mentions 'MAT launcher and Java runtime availability,' which hints at the purpose of 'mat_home' and 'java_path,' but does not explicitly define their meaning, expected format, or whether they are required. This leaves significant ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Validate MAT launcher and Java runtime availability.' It uses a specific verb ('validate') and resource ('MAT launcher and Java runtime availability'), and it distinguishes the tool from its siblings (which focus on index status, OQL queries, specifications, and reports). This makes the tool's role in the ecosystem obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites, context, or cases where another tool like 'mat_index_status' would be more appropriate. This leaves the AI agent without decision-making support for selecting this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mat_index_statusB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries full burden. It indicates a read-only status check, which is a clear behavioral trait. However, it does not disclose error handling or what happens on invalid input (e.g., nonexistent heap path). This is adequate for a simple tool but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 11 words, front-loading the verb and objective. Every word is meaningful; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity and lack of output schema, the description should specify the return type or format (e.g., boolean, message). It does not, leaving ambiguity about the report output. This is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one required parameter 'heap_path' with no description in the schema (0% coverage). The tool description does not explain what 'heap_path' represents (e.g., filesystem path, URL) or any constraints. This provides no value beyond the parameter name itself.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Report whether MAT index artifacts already exist for a heap dump.' It specifies a verb ('Report'), a resource ('MAT index artifacts'), and the object ('heap dump'). This differentiates it from sibling tools like mat_healthcheck or mat_oql_query, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, or any prerequisites or limitations. It does not mention when not to use it or how it relates to other sibling tools. The usage context is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mat_oql_specA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full weight. It implies a read-only, informative operation, but does not detail any behavioral traits such as data source, caching, or error handling. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no fluff. Clearly conveys the purpose with front-loaded key terms. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and no output schema, the description covers the purpose well. It specifies what is returned (guidance, patterns, limitations), which is sufficient for a spec tool. Minor improvement would be to mention the format of the output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist in the schema, so schema coverage is 100%. The description adds no param info, which is fine. Baseline score 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns MAT OQL parser-mode guidance, supported patterns, and known limitations, with a specific verb 'Return' and a distinct resource. It differentiates from sibling tools like mat_oql_query (which executes queries) and mat_parse_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like mat_oql_query or mat_parse_report. The description lacks context about prerequisites or typical usage scenarios, leaving the agent to infer.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.