Server Quality Checklist

Tool Scores

• summarize_analysisA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden but fails to disclose behavioral traits such as whether the tool is read-only, requires authentication, or has side effects. It only describes the input and output without warning of any limitations or behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured with a clear one-line purpose followed by Args and Returns sections. However, the parameter list is somewhat formulaic, and the Returns section is minimal.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema (not shown), the description's minimal return statement is acceptable. Parameter descriptions are complete for basic use, but the tool's purpose is fully covered. It is complete enough for the given complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description provides clear semantic meanings for all 6 parameters in the Args block (e.g., 'Total events processed' for n_events_processed). This adds value beyond the schema, though some details like allowed values for 'sign' are missing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Generate a summary of analysis results for a dataset.' which is a specific verb ('generate') and resource ('summary of analysis results'), and it clearly distinguishes from sibling tools like 'list_available_datasets' or 'get_dataset_info'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, no prerequisites, and no exclusions. It only states the basic function.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dataset_infoA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description mentions return fields (file count, size, availability) but does not disclose if the operation is read-only, requires authentication, or has any side effects. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is concise, structured with Args and Returns sections, and front-loaded with purpose. No extraneous text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists (not shown but indicated), so description doesn't need to detail return format. However, usage guidelines are missing, and parameter semantics could be slightly richer. Overall fairly complete for a simple info tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but description adds examples for defname and lists allowed values for location (disk, tape, scratch, nersc), which adds meaning. However, it doesn't mention the default value for location.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Get information about a specific SAM dataset definition' with specific verb and resource. It provides example parameter values and distinguishes from sibling tools like list_available_datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings. For instance, it doesn't contrast with 'list_available_datasets' or suggest scenarios for querying a specific definition.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_available_datasetsA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description only states return type, lacking details on idempotency, authorization, or side effects. As a read-only list, it could note that it is safe to call repeatedly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler: states purpose and return value. Highly efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a parameterless tool with an output schema; clearly identifies what is listed and the return format. No missing elements for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, so schema coverage is trivially 100%. Description adds no param info since none exist, earning baseline of 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Describes listing available SAM dataset definitions for Mu2e analysis, with a clear verb and resource; distinguishes from sibling tools like get_dataset_info or create_dataset_filelist.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage for browsing datasets before using other tools, but no explicit guidance on when to use or not use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_dataset_filelistA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions creating a filelist and using a temp path if output_path is None, but does not disclose side effects (e.g., file system writes, disk usage), required permissions, or potential risks such as overwriting existing files.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with a clear opening line and labeled args/returns sections. It is concise without being terse, though minor redundancy (e.g., 'prepares a dataset') could be trimmed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and empty schema descriptions, the description covers the tool's core purpose, parameters, and return format. It lacks explicit usage guidance and behavioral details, but the presence of an output schema offsets the need to explain return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions are empty (coverage 0%), so the description provides critical meaning: defname's example, location's allowed values, max_files as testing limit, and output_path's fallback behavior. This adds substantial value beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action (Create), resource (filelist from SAM dataset), and context (for analysis with refana tools). It distinguishes from sibling tools like list_available_datasets or get_dataset_info by focusing on creation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states it prepares a dataset for refana analysis tools, implying usage context, but does not explicitly guide when to use this tool versus alternatives or mention when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.