Server Quality Checklist

Tool Scores

• detect_anomaliesA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It details the three anomaly checks, severity levels, and output structure (Findings with supporting_rows and total_supporting_count). It also specifies acceptable period column formats. This provides good transparency, though it doesn't mention whether the tool is read-only or has side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph that efficiently front-loads the core purpose and lists the three checks. It is concise without superfluous text, though it could benefit from bullet points for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and low schema coverage, the description covers the tool's functionality and output structure well. However, it omits parameter descriptions for three of four parameters, making it incomplete for an agent to correctly select and invoke the tool without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description should explain each parameter. It only addresses the period_column format in the last sentence, providing no guidance on file_path, account_column, or amount_column. This leaves most parameters inadequately explained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs three specific anomaly checks on NetSuite GL-style exports and returns Findings. It lists each check with severity and description, making the purpose very clear and distinguishing it from sibling tools like aggregate_export or query_export.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains what the tool does but does not explicitly state when to use it versus alternatives like categorize_by_memo or get_headers. It implies the tool is for anomaly detection in financial data, but lacks direct guidance on prerequisites or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_exportsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and discloses error behavior (unparseable files return parse_error). It mentions return fields but does not address authorization or read-only nature. The disclosure is useful but could be more comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no waste: first sentence covers purpose and output, second sentence covers error handling and usage recommendation. Every sentence adds value, and the structure is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the main functionality, error case, and usage hint. An output schema exists, so return value details are not needed. For a simple list operation with one parameter, it is nearly complete, though additional context about directory path might help.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'directory' lacks any description in the schema (0% schema description coverage). The description mentions 'under NSMCP_ROOT' but does not clarify if it is relative or absolute, format, or valid values, leaving the agent with insufficient semantic guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists NetSuite saved-search exports under NSMCP_ROOT, specifies the return type (ExportSummary with fields), and distinguishes itself by recommending it as the first call when exports are unknown. This is a specific verb+resource with clear sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Call this first when you don't already know which exports are available,' providing clear usage context. However, it does not explicitly state when not to use this tool or mention alternative sibling tools (e.g., query_export), but the guidance is still effective.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• categorize_by_memoA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the burden. It explains the matching logic and fallback to 'Uncategorized', and that it returns tagged rows with counts. However, it does not explicitly state that the tool is read-only or if it modifies the original file. It also doesn't mention error handling or required permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences, each serving a purpose: the first states the core function, the second gives domain context, and the third explains the rules and output. No wasted words, and the most critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists (as per context), the description is complete enough. It covers the input parameters, matching behavior, and output shape (tagged rows + per-category breakdown). It assumes reasonable domain knowledge about exports and rows but provides sufficient detail for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description compensates by explaining the roles of all three parameters: file_path (via export context), memo_columns (pass both main and line), and rules (maps category to keyword list with first-match wins). This adds significant meaning beyond the bare schema, though it lacks examples or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: tagging rows with a category based on case-insensitive substring matches across memo columns. It specifies the verb (tag), resource (rows from a NetSuite GL export), and method (keyword matching). The purpose is distinct from sibling tools like aggregate_export or detect_anomalies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context for when to use the tool, e.g., for NetSuite GL exports with memo columns. It implies usage for categorization tasks but does not explicitly compare to siblings or state when not to use it. The guidance is clear enough for an AI agent to infer appropriate use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_headersA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes return values and warns about other tools' error behavior. No annotations provided, but description covers key behavioral aspects. Could mention read-only nature explicitly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no fluff. Front-loaded with functionality and usage advice.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given tool simplicity (one param, likely simple output), description covers purpose, usage context, and return values adequately. Output schema existence reduces burden.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only parameter 'file_path' not described beyond name; schema coverage is 0%. Though context implies it's a file path to an export, explicit description would improve clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Explicitly states it returns column headers with spreadsheet letters and header_row. Clearly distinguishes from siblings by advising to call it before query_export or aggregate_export when column names are unknown.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: use when you don't know column names, as other tools error on typos. Describes the purpose in context of sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_parse_warningsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses on-demand parsing behavior and lists warning kinds. With no annotations, this is good coverage, though it could mention error cases or performance impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with a concise list of warning kinds. Every sentence adds value with no waste. Well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description fully covers what the tool does, when to use it, and what the returned warnings represent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema coverage at 0% and a single trivial parameter (file_path), the description merely restates the parameter without adding meaning. For a simple parameter, this is adequate but not exemplary.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it returns parse warnings for an export at a given file_path, with on-demand parsing. Distinguishes itself from sibling tools like aggregate_export or query_export by focusing on warning details.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to call this tool after another tool reports a non-zero warning_count, providing clear when-to-use context and implied alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aggregate_exportA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, description discloses return order and output naming convention. However, it omits potential side effects, performance implications, or constraints (e.g., file size limits).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences front-loaded with core action, no redundant text. Essential details packed without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (aggregation with grouping and measures) and presence of output schema, description covers key aspects: measure definition, ordering, and use-case differentiation. No critical gaps apparent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds crucial detail: measures structure (column, op, optional alias) and default output key pattern. Does not explain file_path format or group_by semantics, but compensates well for schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it groups rows and computes aggregations per group, listing supported operations (sum, count, avg, min, max) and alias behavior. It explicitly distinguishes from sibling tool query_export, making purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises to use this tool instead of query_export when summary statistics are needed. Provides context on return order (first-seen) and output key defaults, enabling correct tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_exportA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses key behaviors: predicates are AND-combined, operators are listed with defaults (case-insensitive, inclusive date ranges), limit behavior (implicit 1000, limit=0 for total count), and return structure (rows, total_matched, truncated flag). No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences efficiently cover purpose, operators, projections, and limit/return details. Every sentence adds value without redundancy. Front-loaded with the main action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of predicates and limit options, the description covers all essential behavior. The output schema is noted but not needed to explain returns since the description already lists rows, total_matched, and truncated flag. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description adds critical semantics: how filters combine, operator behaviors, default case-insensitivity, limit=0 behavior, and column projection. This goes well beyond what the raw schema provides, earning a top score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as filtering rows from a NetSuite export using a list of AND-combined predicates. It specifies the resource (rows from export), the action (filter), and key details like operators and result. This distinguishes it from sibling tools like aggregate_export or get_headers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool (to filter rows) and provides context on predicates and projections. However, it does not explicitly state when not to use it or mention alternatives among siblings, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.