Server Quality Checklist

Tool Scores

• get_histogramA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses output (per-bucket counts, median/p95/p98) and the raw flag's effect, plus lossless bucket preservation. It doesn't mention permissions, rate limits, or failure behavior, but for a read-only get operation this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact (three sentences) and front-loaded with the core purpose. It avoids redundancy and flows logically from purpose to behavior to raw mode, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 14 parameters and no annotations, the description is not fully complete. It relies on the output schema for return values and on get_summary for behavioral context, but it doesn't describe which parameters are needed or how they interact. Still, it covers the main query type and output structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It only explains raw=True and vaguely references date_comparator. The other 12 parameters (app, timer, date, browser, etc.) are completely unexplained, leaving the agent to guess their meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: providing a histogram distribution with per-bucket counts and percentiles. It names the mPulse query type and distinguishes itself from get_summary by referencing its behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description references get_summary for time selection and drilldowns, giving a clear comparison point for when to use this tool. It also explicitly notes raw=True for untouched JSON. However, it doesn't explicitly state exclusion cases or when to prefer other sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_timersA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the return format (series of {x, y} points), resolution (per-minute), time scope (calendar day or date_comparator window), and data fidelity ('Values are preserved exactly'). This adds significant behavioral context beyond the tool name, though it omits potential edge cases or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the first sentence states the core purpose, followed by usage guidance and return details. No filler or redundancy; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (14 parameters, no annotations, 0% parameter coverage), the description is incomplete. It provides a good high-level overview and output information, but leaves the semantics of all filter parameters undefined, making it hard for an agent to invoke the tool correctly for non-default use cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the 14 parameters. It only briefly mentions 'timer', 'custom_timer', and 'date_comparator' in passing, without details on their meaning, format, or effect. This is a critical gap for a tool with many optional filtering parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a single timer's value by minute over time, forming a time series. It distinguishes itself from siblings by noting it is best for one timer and explicitly referencing get_metrics for multiple timers/metrics, ensuring unambiguous purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Best for one timer... For multiple timers/metrics at once, use get_metrics.' This clearly tells the agent when to use this tool and when to use an alternative, with a specific sibling tool name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_metricsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses return structure (values with history and latest), data timezone, lossless number preservation, and defaults. It also references shared time-selection behavior. While it doesn't discuss permissions or rate limits, it provides substantive behavioral context beyond a minimal statement, so a 4 is warranted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the core purpose, and each sentence adds value: query type, defaults, return format, time behavior, and data precision. There is no redundancy or filler. It is an appropriately sized and well-structured description.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the high complexity (15 parameters, no annotations), the description is only partially complete. It covers the core output and time behavior, and an output schema exists to document returns. However, it does not explain the majority of filter/dimension parameters, nor does it provide an example or detailed drilldown semantics. This is adequate but leaves clear gaps, so a 3 is appropriate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for 15 parameters. It only explains defaults for `metric` and `timer` and mentions `date_comparator` by name. Other parameters like `app`, `browser`, `country`, `percentile`, and `custom_timer` are left completely undocumented, leaving the agent with insufficient information to set them correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb-resource pair: 'Multiple timers/metrics by minute over time.' It further specifies the mPulse query-type, defaults, and return shape, clearly distinguishing it from siblings like get_summary or get_histogram, which are likely aggregate or distribution tools. The phrase 'by minute over time' captures the tool's unique time-series focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it returns per-minute time series data for multiple timers/metrics, and notes that time selection/drilldowns behave like other tools. However, it does not explicitly state when to prefer this tool over alternatives or mention exclusions. This is 'clear context, no exclusions,' which fits a 4.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_query_typesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It clearly states the tool lists known query types with summaries, implying a read-only discovery operation, and adds useful dynamic context ('this server knows about') and relationships to sibling tools. It does not discuss auth or rate limits, but those are not critical for a low-risk list operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two compact sentences with no filler. The first sentence states the core function, and the second provides useful sibling-tool context in a structured, readable way. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (zero parameters) and the presence of an output schema, the description is complete. It explains what the tool does, what it returns (one-line summaries), and how it relates to the other query tools, which is sufficient for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and schema coverage is 100%, so the baseline is 4. The description adds no parameter-specific information because none is needed; there are no parameters to explain.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb ('List') and a clear resource ('mPulse query-types this server knows about'), and adds that it returns one-line summaries. It also distinguishes this listing tool from sibling query tools by explicitly mapping query types to get_summary, get_histogram, get_timers, and get_metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that this tool is for discovering what query types exist, but it does not explicitly say 'use this when you need to discover available query types' or provide direct guidance on when to choose it over siblings. It does clarify that explicit tools cover certain types and everything else goes through the generic query tool, which is useful context but not a direct usage guideline for list_query_types.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_queryA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of disclosure. It explains that the tool returns parameters, drilldowns, response shape, and caveats, which gives a clear picture of what to expect. It does not list specific caveats, but the mention of them is useful and aligns with the 'describe' nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose and immediately followed by usage instructions. Every word is necessary, and the use of backticks for code and examples is clean and concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the one-parameter tool and the presence of an output schema, the description adequately covers what the tool does and how to source the parameter. It mentions caveats, hinting at possible edge cases. It doesn't repeat return value details (handled by output schema), so it is complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the tool description compensates by explaining that query_type is a slug from `list_query_types` and provides examples. This adds meaningful meaning beyond the bare schema, though a full list of valid slugs would be even better.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Describe a query-type' with specific details on what is described (parameters, drilldowns, response shape, caveats). It distinguishes itself from sibling tools like get_summary or query by focusing on describing query types rather than executing them or retrieving data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Pass a slug from `list_query_types`' with concrete examples. This implies a prerequisite workflow and gives clear context on how to use the tool, though it does not explicitly mention when not to use it or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and delivers useful behavioral traits: `format=json` is added automatically, the one-calendar-day-per-query constraint is mentioned, and `raw=True` returns untouched JSON. It does not cover error behavior or authorization requirements, but for a generic query tool this is substantial context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured, starting with the core purpose, then usage guidance, parameter details, and a final note. Every sentence adds value; there is no fluff or repetition. Appropriate for a tool that needs this level of context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex and arbitrary, but the description gives enough context for a knowledgeable agent: examples, constraints, and the raw option. It also references `list_query_types` for discovering types. It falls slightly short by not elaborating on the `app` parameter and the one-calendar-day constraint, but overall it is quite complete given the output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does so effectively for the tricky parameters: `params` is explained as a dict of hyphenated wire names with a concrete example, and `raw` is explained. However, the `app` parameter is left undocumented, and `query_type` is only implied by name. This is a strong but not complete compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Call an arbitrary mPulse query-type with arbitrary parameters.' It uses a specific verb ('call') and resource ('query-type'), and distinguishes itself from sibling tools by explicitly targeting query-types 'not covered by the explicit tools' with concrete examples.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use this tool: 'Use this for query-types not covered by the explicit tools (see `list_query_types`)'. It also points to the alternative (explicit tools) and provides examples, giving both a clear when-to-use and an implied exclusion for covered types.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_appsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. The verb 'List' implies a read-only operation, and it adds useful behavior about the default app fallback. It does not explicitly state whether any side effects exist, but for a zero-parameter list command, the implied read-only nature is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short sentences, each adding distinct value: purpose, usage timing, and parameter context. The key information is front-loaded, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool with zero parameters and an output schema, the description fully covers what the tool does and when to use it. It also integrates well with the sibling tools by explaining the optional `app` argument pattern, making the description complete in context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is nothing about this tool's schema to clarify. The description adds valuable context about the default app and the `app` argument on sibling tools, which helps the agent understand the ecosystem, but that is not directly about this tool's parameters. Baseline for a zero-param tool is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' with an explicit resource ('registered mPulse apps and the default app'), making the tool's purpose immediately clear. It is inherently distinct from sibling analytic tools like get_summary or get_metrics, as it returns app identifiers rather than metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to 'use this before other tools' so that real app names can be referenced. It also clarifies the behavior of the `app` argument across other tools, which is important context for when this listing tool should be called.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_summaryA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden. It discloses per-minute aggregation over a single calendar day, timezone default, raw=True behavior, no numeric rounding, and the non-error behavior for unsupported drilldown combos. This is rich, truthful context beyond any structured metadata.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but well-structured with line breaks, bullet-like sections, and a logical flow from stats to time selection to timer to drilldowns to raw mode. Every sentence adds value; the length is justified by the 18-parameter complexity and lack of annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with no annotations and zero schema descriptions, this description covers time selection rules, timer choice, percentile range, drilldown behavior, raw mode, timezone, aggregation granularity, and error behavior. Since an output schema exists, return values are already documented, so the description doesn't need to over-explain them.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates by explaining key parameters: date, date_comparator, timer/custom_timer, percentile, raw, and several drilldowns (page_group, browser, country, device_type, ab_test). It does not explicitly explain every parameter (e.g., os, region, connection_type), but the pattern and examples provide enough guidance for most.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Aggregate summary stats for one timer (median, margin-of-error, count, p95, p98).' This clearly distinguishes it from siblings like get_histogram and get_metrics by naming the exact aggregate outputs and the 'one timer' scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states that exactly one time selector (date or date_comparator) is required, that date ranges are unsupported and must be split into per-day calls, and that drilldowns filter data with unsupported combinations returning empty data + note instead of an error. This gives clear when-to-use and constraint guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.