Server Quality Checklist

Tool Scores

• update_alertB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It discloses that the tool pre-checks edit access and refuses if can_write is false, and requires docsConsulted=true. However, it does not describe what the update does (e.g., merge vs replace fields), whether it errors on non-existent alerts, or what the return value looks like. It adds some value but leaves significant behavior unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no wasted words. The purpose is front-loaded, followed by key preconditions. Every clause earns its place, and the documentation-first policy is stated clearly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a complex tool with 15 parameters and no output schema. The description covers purpose and preconditions but does not explain update semantics (partial vs full replacement), success indicators, error handling beyond access refusal, or what happens to unspecified fields. Given the complexity, an agent would need more guidance to call it correctly without guessing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with every parameter having a description. The description reinforces the docsConsulted requirement but adds no new meaning beyond the schema's 'Must be true.' The behavioral notes about access control do not pertain to parameters. The baseline of 3 applies since the schema already documents all parameters thoroughly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Update an existing alert.' This clearly distinguishes it from create_alert (since it says 'existing') and get_alert (since it says 'update'). It doesn't explicitly name sibling alternatives, but the purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the verb 'update' and 'existing alert', but it does not explicitly say when to use this tool versus create_alert or other siblings. The documentation-first directive is a how-to guideline, not a tool-selection rule. Preconditions (edit access check, docsConsulted) are behavioral constraints, not usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_alertsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It does disclose that results are limited to what the caller 'can see' and defines alert classification as 'scheduled + triggering,' which adds useful context. However, it does not explicitly state read-only behavior, authentication expectations, or whether only owned alerts versus all visible alerts are returned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence with no filler. Every phrase earns its place: the verb, the resource, the alert classification, and the visibility qualifier. It is appropriately compact for a straightforward list tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a simple list operation and the schema fully documents all four parameters. However, with no output schema and no annotations, it does not mention sibling routing (e.g., using get_alert for details) or any additional behavioral context, leaving moderate gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and each parameter already has semantic meaning in the schema. The description adds no parameter-level detail beyond the core listing concept, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'List saved searches classified as alerts,' and adds clarifying scope with '(scheduled + triggering)' and 'that you can see.' This clearly separates it from siblings like list_dashboards, list_reports, and get_alert without requiring schema inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives such as get_alert, create_alert, update_alert, or list_reports. The only usage signal is the verb 'List,' so the agent must infer the appropriate context rather than being told.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_reportA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description adds useful behavioral context: reports are scoped to YOUR profile, cron scheduling is optional, and docsConsulted must be true. However, it does not disclose whether authentication is required, what side effects occur on duplicate names, or what the response contains.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The purpose is front-loaded and the remaining sentences quickly cover scheduling, the docs gate, and the documentation-first rule. 'or a dashboard definition' is slightly tangential for a report-creation tool, but overall the description is compact and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The schema plus description give enough to invoke the tool with the required name, search, and docsConsulted fields. But there is no output schema and no mention of authentication or return behavior, so the description is not fully complete for a 9-parameter write operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the detailed property descriptions already carry most of the parameter meaning. The description only restates the cron option and the docsConsulted requirement without adding new semantics for the other parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Create a report (saved search) under YOUR profile.' The parenthetical 'saved search' and the personal profile scope clearly distinguish it from sibling tools such as create_dashboard and create_alert.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It mandates a docs-first workflow and the docsConsulted=true gate, but it never names alternatives or states when not to use this tool. The intended use is implied by the tool name and 'report (saved search)' rather than explicitly contrasted with create_alert or create_dashboard.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_alertA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. While it lists the returned fields, it does not explicitly state whether the operation is read-only, requires special permissions, or has any side effects. The verb 'Get' suggests read-only behavior, but that is not explicitly disclosed, and there is no mention of error behavior or access requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the core purpose ('Get one alert') and efficiently lists the returned fields in a compact colon-separated list. No unnecessary words or repetition; every element adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple getter tool with no output schema and no annotations, the description is largely sufficient: it clearly states the resource type and enumerates the return fields. However, it omits any mention of permissions or error handling, which might be expected in a Splunk context, though these may be implicitly understood from the environment.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter (app, name, owner) already documented in the input schema. The description adds minimal value beyond the schema, only clarifying that the 'name' refers to a saved search with alerting. Since the schema carries the parameter details, this is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and the resource 'one alert', and clarifies that an alert is a 'saved search with alerting'. It distinguishes from list_alerts by focusing on a single item and enumerates the specific fields returned (SPL, schedule, trigger condition, actions, ACL), leaving no ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for retrieving a single alert, which naturally differentiates it from list_alerts (plural). However, it provides no explicit statement of when to use this tool over alternatives, nor any exclusions or conditions. The usage context is inferred but not made explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dashboardA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full disclosure burden. 'Fetch' implies a read-only operation, and it usefully lists what is returned. However, it does not explicitly state that the operation is non-destructive, note auth requirements, or describe error behavior (e.g., dashboard not found).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single front-loaded sentence that leads with the core purpose before listing return contents. Zero filler—every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description adequately enumerates what the tool returns (definition, sharing metadata, format, can_write flag). For a simple fetch tool whose params are fully schema-documented, this is largely sufficient; only error handling and explicit read-only status are unaddressed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all three parameters (app, name, owner) are already documented. The description adds little parameter-specific meaning beyond the schema, mentioning app/owner only as part of the return value rather than as call guidance. Baseline 3 is appropriate when the schema carries the load.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Fetch'), a specific resource ('one dashboard's full definition'), and enumerates the return contents (format, app/owner/sharing, can_write flag). This clearly distinguishes it from siblings like list_dashboards (enumeration), check_dashboard_access (access check), and create/update_dashboard (mutations).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'one dashboard's full definition' implies this tool is for retrieving a single dashboard's complete definition rather than listing or checking access, but no alternatives are named and no exclusions given. Usage context is inferred rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_reportA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden; it does communicate that the tool returns a report's configuration rather than just existence. It does not explicitly state read-only behavior, permissions, or behavior when the report is missing, leaving some ambiguity for a tool with no annotation safety hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with no filler; the key verb and object lead, and the additional content list is compact and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool with no output schema, the description covers what the agent receives (SPL, schedule, ACL, config). It lacks explicit usage differentiation from list_reports, but the schema fleshes out the parameters and the rest is straightforward.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the parameter descriptions already explain app, name, and owner. The tool description adds no parameter-level meaning beyond labeling the target as a saved search, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb (Get) and resource (one report), disambiguates 'report' as a saved search, and enumerates the returned config (SPL, schedule, ACL, full config). This clearly distinguishes it from siblings like list_reports and create_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'one report' implies use with list_reports for enumeration, but there is no explicit when-to-use guidance or mention of alternatives. An agent must infer that this is the detail-fetch variant based on sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• splunk_docsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states what the tool returns (documentation references) and the mandatory ordering ('Call this first'), but it does not disclose side effects (though it implies read-only), authentication requirements, rate limits, or the exact format of the returned references. It is not contradictory, but it is not rich in behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. The core purpose and directive are front-loaded in the first sentence, with the optional parameter detail in the second. It is concise, well-structured, and immediately actionable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with a single optional parameter and no output schema. The description communicates its purpose and invocation order, but it does not clarify what the 'documentation references' actually contain (e.g., links, text snippets, file paths) or how they will be presented. Given the lack of an output schema, this leaves some ambiguity for the agent, but the core context is sufficient for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema property 'topic' already includes a description listing the valid options ('Optional focus: spl, studio, classic, or rest. Omit for all references.'). The tool description repeats the same information ('Optional topic: spl | studio | classic | rest.'). Since schema coverage is 100% and the description adds no new semantic meaning beyond the schema, the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Return the official Splunk documentation references') and the resource being returned (documentation references). It also specifies the purpose ('MUST consult before authoring SPL or dashboards'), which distinguishes it from sibling tools that actually create or query data. This is unambiguous and well-scoped.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to call it ('Call this first when writing queries or dashboard definitions'), providing clear context for usage. However, it does not mention any alternative tools or exclusion scenarios (e.g., 'do not call if you already have the docs'), so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_dashboardsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that visibility is enforced by the user's account, which is useful behavioral context. However, it doesn't mention pagination behavior beyond the count parameter, sorting, or what happens when no dashboards match. The description adds some value but leaves gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no waste. The core purpose is front-loaded, filters are listed compactly, and the wildcard note is a useful addition. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list tool with no output schema, the description covers the essential behavior: what it lists, visibility constraints, and filtering options. It doesn't describe the return format, but for a list tool this is less critical. The main gap is lack of explicit pagination or sorting behavior, but the count parameter covers the main need.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters. The description adds context about wildcards and case-insensitive substring matching, which is helpful, but it doesn't significantly go beyond the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists dashboards visible to the user, with explicit filtering by app, owner, or name/label substring. It distinguishes itself from sibling tools like get_dashboard (which retrieves a single dashboard) and create/update_dashboard (which mutate dashboards).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use this tool: to list dashboards the user can see, with filters. It doesn't explicitly name alternatives or state when not to use it, but the context of sibling tools and the clear filtering guidance make the usage context strong. It could be improved by explicitly contrasting with get_dashboard or check_dashboard_access.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_dashboardA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses two significant behaviors: it pre-checks edit access and refuses if ACL denies write, and it requires docsConsulted=true. These go beyond the schema and are valuable for an agent. It does not mention replacement semantics (that the definition replaces the entire existing one) or what the success response looks like, which are gaps, but the disclosed behaviors are substantive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the primary purpose. It follows a logical order: purpose, behavioral trait, requirement, and policy. Each sentence adds value, though the documentation-first instruction is somewhat verbose and could be tightened. Overall, it is well-structured and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, no output schema, no annotations), the description is not fully complete. It does not state that the definition parameter replaces the entire existing definition (a critical semantic), nor does it describe the success return value or other failure modes beyond ACL denial. The access check and documentation requirement are covered, but an agent would still be uncertain about the full scope of the operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description reinforces the docsConsulted requirement ('Requires docsConsulted=true') and the documentation-first policy, but these are already captured in the schema's parameter descriptions. No additional semantic value is added beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Update an EXISTING dashboard's definition,' which is a specific verb and resource that clearly distinguishes it from sibling create_dashboard. The word 'EXISTING' signals the key differentiator, and the focus on 'definition' narrows the scope to the dashboard's content rather than its metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states the tool is for existing dashboards, which implies it is not for creation, and the context of sibling tools (create_dashboard, get_dashboard) reinforces this. However, it does not explicitly name an alternative or provide a 'when not to use' clause, so the guidance is clear but not fully explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• whoamiA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral transparency. It discloses that the tool returns the current username and notes a key behavioral implication: new dashboards are created under this profile. This adds context beyond the name, but it does not describe the response format, potential errors (e.g., if not authenticated), or any side effects. Since it's a read-only identity query, the information provided is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise – two short sentences that pack key information: the primary purpose and a critical contextual implication. Every word serves a purpose, and the most important information is front-loaded. No redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no output schema, and a single purpose), the description is nearly complete. It tells the agent what the tool returns (username) and why it matters (dashboard creation context). The only minor gap is that it doesn't specify the exact format of the output or behavior when no session exists, but this is a minor omission for such a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema coverage is 100% (vacuously). Since there are no parameters to describe, the description does not need to add parameter details. The baseline for zero-parameter tools is 4, and the description appropriately focuses on the tool's purpose and behavior, earning a solid score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to show the Splunk username for the current session. It uses a specific verb 'Show' and resource 'Splunk username'. It also adds relevant context about new dashboards being created under this profile, which helps the agent understand the practical implication.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly conveys when to use this tool – when you need to know the current username, particularly before creating dashboards. However, it does not explicitly mention when not to use it or compare it to alternatives. Given the sibling tools include splunk_login and various create tools, the usage context is implied but not fully developed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_dashboard_accessA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. 'Reads the Splunk ACL and reports can_write' makes the read-only nature and return style explicit. It could mention error or login preconditions, but the tool is a simple access check and no side effects are implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no filler. The key instruction appears first, and the second sentence states what the tool reads and returns. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, has full schema coverage, and no output schema. The description gives enough return semantics ('can_write' plus a verdict). It does not specify behavior on missing dashboards or ACL errors, but it is sufficient for the tool's narrow access-check purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the app, name, and owner parameters are already documented. The description adds workflow context but not additional parameter-level meaning, which is acceptable given the schema handles it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb and resource: check whether you can edit a dashboard. It also names the mechanism (reads the Splunk ACL) and the output (can_write with a plain-language verdict), which distinguishes it from update_dashboard and get_dashboard.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent to run this 'BEFORE trying to update it,' making the timing relative to update_dashboard clear. It does not enumerate all alternative tools or with exclusions, but the intended workflow is obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_alertA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the mandatory docsConsulted flag and the documentation-first workflow, which is useful, but it does not describe side effects of creation (e.g., whether existing alerts with the same name are overwritten), authentication requirements, or error behavior. The 'read the official Splunk docs ... or a dashboard definition' snippet is generic and slightly off-topic for an alert tool, but the core constraint is conveyed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no fluff. The purpose and scope are front-loaded, the prerequisite is stated in the second sentence, and the third gives an actionable directive. Every sentence earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 14-parameter tool with no output schema or annotations, the description provides key usage guidance (docs-first, docsConsulted requirement) and a conceptual model. It does not mention authentication (though splunk_login sibling exists) or success/error output, but schema covers parameter defaults and enums. The slight mismatch about 'dashboard definition' is a minor flaw, but overall the agent has enough to call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds conceptual meaning beyond the schema by explaining that an alert is a scheduled saved search with a trigger condition, which clarifies how parameters like search, condition, threshold, and cron relate. It also emphasizes that docsConsulted must be true, reinforcing its semantics beyond the schema's simple 'Must be true.'

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (create), the resource (alert), and defines it as a 'scheduled saved search with a trigger condition,' which immediately distinguishes it from sibling tools like create_report or create_dashboard. It also adds scope ('under YOUR profile'), removing ambiguity about the target user.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states a prerequisite (docsConsulted=true) and gives a direct instruction to use the splunk_docs tool before writing SPL, preventing guesswork. It does not explicitly list when not to use it (e.g., for updating an existing alert), but the creation context is clear, and the 'under YOUR profile' constraint implies personal alerts rather than shared ones.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_dashboardA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden and covers important behavior: it is a mutating create operation, the dashboard is owned by the caller's Splunk user, the default visibility is private, and documentation consultation is mandatory. It doesn't detail failure modes or overwrite behavior, but 'NEW' implies non-overwrite and the core side effects are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three front-loaded sentences with no filler. The core purpose, key constraints, supported formats, and documentation-first requirement are all stated directly and efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a create action with no output schema, the description covers ownership, privacy default, supported definition formats, and the required docsConsulted flag. It omits only minor details such as the app namespace default, which the schema already documents, so an agent has enough context to invoke this correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline applies. The description reinforces the format mapping (Studio JSON vs Classic XML) and emphasizes docsConsulted=true, but these details are also present in the input schema. Little new parameter-level meaning is added beyond emphasis.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb and resource ('Create a NEW dashboard') and adds distinguishing details: it is scoped to the caller's profile, private by default, and supports both Dashboard Studio JSON and Classic Simple XML. This makes it easy to separate from sibling tools like update_dashboard or create_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly indicates when this tool is appropriate — when creating a new dashboard owned by the calling user — and explicitly instructs the agent to consult Splunk docs first via the splunk_docs tool. It doesn't explicitly enumerate exclusions or contrast with update_dashboard, but the 'NEW' and ownership language provides sufficient routing context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_reportsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the behavioral burden. It communicates that this is a read-style operation, that visibility is scoped to the caller ('that you can see'), and that filtering is optional. It does not discuss ordering or response shape, but it does clarify the tool's main behavioral boundaries.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every clause earns its place: the object, the scope qualifier, the exclusion of alerts, and a one-line pointer to filters. No fluff, examples, or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Coverage is strong given the tool's simplicity: the operation, object type, visibility scope, and filter dimensions are all present. The schema fills in count/query semantics impossibly. Missing only a hint about output shape/ordering, which is minor for a list endpoint.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%; the description restates filter dimensions but adds little beyond the schema. It does clarify that 'name' relates to the query param, but the schema's 'Case-insensitive substring match' and count defaults already carry most of the semantic load.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence names the operation ('List'), the object ('saved searches classified as reports'), and the scope ('that you can see'). The parenthetical '(not alerts)' immediately disambiguates it from a closely related category, so an agent can identify purpose without guessing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear context for when to use the tool: to list report-type saved searches, not alertschers. The filter phrase tells an agent which selection capabilities exist. It stops short of explicitly naming sibling tools (e.g., list_alerts vs list_reports), so it does not get full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_reportA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses important behavioral traits: it pre-checks edit access and refuses if can_write is false, and it requires docsConsulted=true. It also mandates documentation-first behavior. Since no annotations are provided, the description carries the full burden, and it does so well. It doesn't mention reversibility or side effects, but the pre-check and refusal behavior are significant disclosures. A 4 is appropriate because it goes beyond a simple 'update' statement but doesn't cover all potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the core action: 'Update an existing report.' It then adds critical behavioral constraints in a compact manner. The documentation-first instruction is a bit long but necessary. It earns a 4 because it's efficient and well-structured, though the documentation-first sentence could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (10 parameters, no output schema, no annotations), the description covers the essential behavioral context: pre-checks, refusal condition, and documentation requirement. It doesn't explain return values, but there's no output schema, so the description isn't required to. It could mention what happens on success or failure, but the refusal condition is a key part. A 4 is appropriate because it's complete enough for an agent to call it correctly, with minor gaps around post-update behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 10 parameters. The description adds context for docsConsulted (must be true) and docsReference (the doc URL/section used), which are already in the schema. The description doesn't add much beyond the schema, but the baseline is 3 for high coverage. It does clarify the purpose of docsConsulted and docsReference in the context of the documentation-first rule, which is a slight addition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Update an existing report.' It also distinguishes itself from siblings by mentioning pre-checks and documentation requirements, which are specific to this tool. The verb 'update' and resource 'report' are explicit, and the description adds unique behavioral details that separate it from create_report or get_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool: to update an existing report, and it provides clear prerequisites: pre-checks edit access, refuses if can_write is false, requires docsConsulted=true. It also instructs to read official Splunk docs before writing SPL or dashboard definitions, which is a clear usage guideline. It doesn't explicitly name alternatives, but the context signals and sibling list make it clear this is for updates, not creation or retrieval.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• splunk_loginA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and does well: it reveals browser automation, cookie persistence, SSO-only auth, and troubleshooting steps for IdP/automation failures. It doesn't mention reversibility or that this mutates local state, but the cookie-file note covers the key side effect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences, each earning its place: what it does, how auth works, and what to do on failure. Front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter login tool, it covers behavior, side effects, and failure handling. It doesn't explicitly state return values or confirmation of success, but the description's troubleshooting note ('reported cookie file') implies some status output. Minor gap given no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the schema has nothing to explain. The description's mention of 'reported cookie file' and SSO details adds useful context despite no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb+resource: performs SSO login in a browser and saves cookies for the Splunk REST API. It is immediately distinguishable from sibling query/dashboard/report tools as an auth setup step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes clear it is the SSO-based auth bootstrap for API calls and even gives recovery steps if the IdP redirect or automation blocks the session. It doesn't explicitly say 'run before other tools', but the context strongly implies it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_splunkA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. It discloses that the operation 'runs as you' (authentication context) and that 'Splunk enforces data access' (authorization), which are important behavioral traits. However, it does not explicitly state read-only or potential side effects, though a query inherently implies read-only; it omits details on rate limits or large result handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise yet information-dense: it states purpose, security context, and a documentation-first rule in just two sentences. The structure is logical, leading with the function and then the usage prerequisite.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a query execution tool, this definition covers all essential operational aspects: what it does, how to call it (with parameter details), default values, and a mandatory documentation step. It lacks an output schema but compensates with the maxResults description and is fully adequate for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema describes all 4 parameters with clear explanations, including defaults for earliest/latest and maxResults, and the query parameter notes auto-prefixing of bare terms. This gives agents complete guidance on how to construct inputs without ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run an SPL search and return results', specifying a concrete verb and resource, and adds 'runs as you' which conveys scope. Among siblings, only this tool executes SPL queries, so it is distinct from splunk_docs, dashboard, report, and alert tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to 'Use the splunk_docs tool (topic 'spl') to verify command syntax before composing queries' and emphasizes 'DOCUMENTATION-FIRST' and 'do not guess syntax or schema', giving clear when-to-use and pre-conditions. It routes users to the appropriate alternative (splunk_docs) and clarifies that this tool is for running queries after verification.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.