Server Quality Checklist

Tool Scores

• query_prompt_style_summaryA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It reveals it is a daily rollup but omits behavioral details such as data retention, auth requirements, rate limits, or what 'rule-classified' entails. This is a significant gap for a query tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core purpose. Every word earns its place with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite good schema coverage and no required params, the tool lacks an output schema and the description does not clarify return format, pagination, or limits. For a query tool with 7 parameters, this is a noticeable gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema; it does not explain semantics of parameters like date format, filtering behavior, or the effect of 'all' values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Daily rollup per prompt style (rule-classified)' and explains its utility as 'useful for understanding how users phrase their requests', effectively answering what the tool does and distinguishing it from sibling summary tools like query_language_summary.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a general use case ('understanding how users phrase their requests') but does not explicitly state when to use this tool versus alternatives (e.g., query_language_summary, query_session_summary) or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_organizationsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It only states 'List all' (implying a read operation) but omits details like authentication requirements, potential size of the list, or whether results are paginated. This is insufficient for an agent to infer safe/expected behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at two sentences (20 words). The purpose is front-loaded in the first sentence, and the second adds a critical usage hint. Every word serves a purpose with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  As a simple list tool with no parameters and no output schema, the description provides the basic purpose and a usage hint. However, it does not describe the output format or what fields are returned, leaving the agent to infer. Given the low complexity, this is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters and 100% coverage, so the baseline is 3. The description adds no parameter information because there are none, which is acceptable. No extra meaning is needed beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists 'all beacon organizations (tenants)' available via the configured API. It uses a specific verb ('list') and resource ('organizations') and distinguishes from siblings by focusing on organization discovery rather than queries or configurations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises calling this tool first when the organization is unknown: 'Call this first if you don't know which org to query.' This provides clear guidance on when to use it, though it does not explicitly mention when not to use or provide alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_employee_hourly_summaryA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description lacks disclosure of important behavioral traits such as data freshness, pagination, error handling, cost implications, or what happens with missing dates. The burden is on the description, and it falls short.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the tool's purpose and followed by usage guidance. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the output nature and use cases for a query tool with 7 optional parameters and no output schema. However, it does not describe return format, limits, or behavior when filters yield no results, leaving gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents parameters. The description adds context by explaining the output structure, but does not add parameter-specific meaning beyond what the schema provides, meeting the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns per-user, per-hour usage broken down by tool and model, and provides specific use cases (find power users, peak hours, per-model consumption), distinguishing it from sibling query tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit use scenarios for when to use the tool, but does not mention when not to use or compare directly with alternatives like query_events or query_session_summary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dashboardA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explains the operation (fetch) and return content but does not disclose side effects, permissions, rate limits, or confirm it is read-only. Adequate but not detailed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences effectively convey the tool's purpose and output. No wasted words; front-loaded with the action and key parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description helpfully enumerates the categories of data returned (metrics, activity, traffic, etc.). It covers the main aspects of the tool, though it could hint at the structure or size of the response.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the description adds little beyond what the schema already provides. It mentions 'date range' and 'sub-section' which correspond to from/to and section parameters, but no additional semantic depth.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches the beacon dashboard payload or a sub-section for a date range, listing the specific data components it returns. This distinguishes it from sibling tools like get_session_events or query_* tools which are more narrow in scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for fetching dashboard data but does not explicitly state when to use it versus alternatives, nor does it provide exclusions or prerequisites. The context is clear but lacks explicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_language_summaryA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full burden of behavioral disclosure. It states it is a 'daily rollup' and derives data from 'session language detection', which implies read-only aggregation. However, it does not explicitly confirm non-destructive behavior, describe the rollup logic, or specify what data is excluded. The description is minimal but non-contradictory.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys purpose and use case without wasted words. It is appropriately concise for a tool with 7 parameters already documented in the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema and 7 parameters, yet the description does not explain the return format, aggregation method, or how filters affect the output. For a summary tool, this omission leaves agents uncertain about what data they will receive. The description is too sparse to fully compensate for missing structural metadata.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so each parameter is well described in the schema. The tool description adds no additional parameter-level meaning beyond the schema. Baseline score of 3 is appropriate as the description does not duplicate or subtract value from the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool provides a 'daily rollup per programming language' and its use case ('identify which languages AI assistants are most often working in'). It distinguishes from sibling tools like query_project_summary by focusing specifically on language detection, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear usage context ('identify which languages') but does not explicitly mention when not to use this tool or suggest alternatives among siblings. It is adequate but lacks exclusionary guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_eventsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the default limit (100 rows) and the 'all' flag to disable it, and warns to use sparingly. However, it lacks details on output format, authentication needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the purpose, and contains no fluff. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 9 parameters and no output schema, the description adequately covers the main filters and pagination behavior but omits output specifics and any behavioral caveats beyond the limit.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description mentions filter categories and pagination but does not add significant detail beyond what the schema already provides for each parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Raw event query with date range, project/model/status/user filters and pagination' and specifies it is for inspecting individual events, distinguishing it from sibling summary tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use for inspecting individual events' and lists example uses. It does not explicitly say when not to use, but the context of summary tools in siblings implies differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_project_summaryA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It adds that results are a daily rollup and include counts and token usage, but does not disclose behavioral traits like data freshness, pagination, sorting, or how multiple projects are handled. The transparency is adequate but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading the core functionality and then a usage example. Every word earns its place; no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters and no output schema, the description does not explain return structure or provide sample output. It covers basic purpose but leaves gaps about what the agent can expect in terms of result format or pagination.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description does not add meaning beyond the schema; it only reiterates the project focus. No extra parameter context or examples are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it provides a daily rollup per project with event/request counts and token usage, and gives an example use case. It differentiates from siblings like query_language_summary by specifying 'per project', making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes a concrete usage example: 'Use for 'which project uses the most tokens this week' questions.' It implicitly guides when to use this tool over others focused on different summaries, but lacks explicit when-not-to-use or alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_session_summaryA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully convey behavioral traits. It mentions a rollup but does not specify the exact aggregation method (e.g., totals, averages), sorting order, or pagination details. The vague 'etc.' further limits transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the key purpose, and contains no irrelevant information. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (7 parameters, no output schema), the description is thin. It does not explain what 'heaviest sessions' means exactly, nor does it mention sorting or rollup granularity. It is adequate but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the description does not need to repeat parameter definitions. However, it adds no extra meaning beyond what the schema already provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a per-session rollup focusing on token usage, models, and timestamps. It distinguishes from the sibling tool get_session_events by indicating it is used for initial exploration before drilling in.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use the tool: 'Use to find the heaviest sessions before drilling in with get_session_events.' This provides clear context and an alternative, earning top marks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• health_checkA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It states it checks reachability and health, implying read-only, but doesn't clarify response format or error behavior. Adequate for a simple tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no unnecessary words. Front-loaded with action and resource. Efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Simple tool with one optional param and no output schema. Description covers purpose but omits return format. However, given low complexity, it is fairly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter 'org' with 100% schema description coverage. The description adds minimal value by echoing 'optional' but no extra semantics. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it checks beacon API reachability and optional org health. The verb 'check' and resources are specific, distinguishing it from sibling tools like get_config or query_*.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage as a health check. No explicit when-not or alternatives, but sibling list doesn't contain similar health tools, so context is clear. Could benefit from stating it is safe to call frequently.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_session_eventsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It implies a read operation but does not explicitly state read-only behavior, permissions, rate limits, or side effects. Adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first describes function concisely, second gives usage guidance. No redundant words, front-loaded with key purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description lists return elements (prompt previews, token usage, errors, model) adequately. Usage context with sibling tool is provided. Missing full details on pagination, error handling, or format, but sufficient for a drill-down fetch tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description does not add new parameter-level detail beyond what the schema already provides; it only mentions 'single session' contextually.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses specific verb 'Fetch' and resource 'full event chain for a single session', lists return elements (prompt previews, token usage, errors, model), and distinguishes itself from sibling 'query_session_summary' by positioning as a drill-down tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'Use after query_session_summary to drill into the heaviest sessions,' providing clear context and purpose, making it easy for an agent to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_configA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the read-only nature ('Read'), the type of config ('public beacon'), and specific contents ('min session event count, model pricing'). It does not mention side effects, auth, or rate limits, but for a simple read operation this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no redundant words. It front-loads the key information and earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool with one parameter (100% schema coverage) and no output schema, the description adequately explains the purpose and the type of data returned. It is complete for an agent to select and invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% coverage with one param 'org' described as 'Organization to read config for.' The description adds value by specifying that the config is for an org and what data it contains, going beyond the schema's basic description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Read' and the resource 'public beacon configuration', and specifies the contents (min session event count, model pricing). It is distinct from siblings like 'get_dashboard' and 'list_organizations'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for a specific org ('for an org'), providing clear context. However, it does not explicitly state when not to use it or mention alternatives, though it is straightforward given the sibling tool names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.