Server Quality Checklist

Tool Scores

• gi_whoamiA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint and openWorldHint. The description reinforces this and adds behavioral context: it explains the tool confirms API key validity, lists reachable organizations, returns their IDs, and how to use the output to enable validation runs. It doesn't cover failure modes or rate limits, but key behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly three sentences: function, safety/usage, and output guidance. Every sentence adds value, is front-loaded, and has no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter diagnostic tool with no output schema, the description covers what it does, when to use it, what it returns, and how to process the return value (export to GHOST_INSPECTOR_ORG_ID). It is complete enough for an agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and schema coverage is trivially 100%. The description adds context about the implicit API key and its use, but there are no parameter-level details to explain. Baseline for zero-parameter tools is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb+resource: 'Confirms the configured API key works and lists the organizations it can reach.' It clearly distinguishes from sibling tools by focusing on credential verification and organization access, which is the core of 'whoami'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'safe to call first when diagnosing setup,' providing clear when-to-use context. It doesn't explicitly list alternatives or exclusions, but the positioning as a first diagnostic step implies when to prefer this over sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gi_module_usageA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite having readOnlyHint and openWorldHint annotations, the description goes far beyond them by disclosing performance characteristics (one request per test at low concurrency), edge cases (import-only tests, dangling execute steps, vacuousTests), and error handling (unreadable definitions counted). It also surfaces surprising behaviors like vacuous tests passing silently, which is critical context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is lengthy but densely packed with high-value information. It is front-loaded with the core purpose and then expands into important behavioral and edge-case details. The structure uses clear paragraphs and an emoji marker for emphasis, but it could potentially be tightened without losing meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and lack of an output schema, the description thoroughly explains what the tool returns (reverse index, importer lists, blast radius), flags important edge cases, and provides operational guidance. It covers performance, error semantics, and even conditions that make the tool necessary, making it highly complete for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage for the single optional 'module' parameter, including its case-insensitive substring matching and its effect on capping. The description itself adds minimal parameter-level detail beyond what the schema already documents, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific, action-oriented question - 'if I edit this module, which tests break?' - and clearly states it builds a reverse index of execute steps. It distinguishes itself from sibling tools by targeting module import relationships and blast radius, not just inventory or stale-test detection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to 'Run this BEFORE editing any module' and advises to 'Call it once and work from the result rather than per module.' It also warns that this is the expensive tool and explains the cost implications, giving clear usage context without needing to compare to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gi_inventoryA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes well beyond the readOnlyHint annotation by explaining the import-only module counting behavior, the fact that marking a test import-only deletes stored results, and the interpretation pitfall of modules appearing permanently unrun. It also discloses the ~440 KB fetch and account-wide totals, adding significant behavioral context that annotations do not cover.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by a critical caveat and a performance/usage note. Each paragraph serves a distinct function, and no sentence is redundant or filler, achieving high information density without being bloated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (two optional params, no output schema), the description covers the essential return contents (counts, failing names, notes field), the behavioral caveat around import-only tests, and practical guidance on fetch size and filter semantics. This is sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already fully documents both parameters (folder and failingOnly), so the baseline is 3. The description adds meaningful nuance by clarifying that even when the 'folder' filter narrows the listing, the reported totals remain account-wide, which affects interpretation of both parameters. It also implicitly advises against using 'folder' to probe piecemeal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb phrase ('Read-only tour of the whole account') and enumerates exactly what is included (folders, suites, per-suite counts, failing names). This clearly distinguishes it from other account-related tools like gi_whoami or gi_stale_tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Start here' and explains that no other question about this account can be answered without this overview, making the primary use case clear. It also advises against probing folder by folder because of the fetch size. However, it does not explicitly name alternatives or say when not to use this tool, so it falls short of the highest bar.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gi_stale_testsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes far beyond the readOnlyHint annotation, disclosing cost ('Costs one request per test, a few seconds for a few hundred tests'), algorithmic details (walking the whole module chain), and side-effect warnings about test re-runs. It also explains exclusions and the additional report of passing-but-unverified tests.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but each sentence earns its place, covering trigger condition, failure categorization, warnings, exclusions, and cost. It is front-loaded with the key instruction and structured logically, making it dense but efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's analytical complexity and the absence of an output schema, the description thoroughly explains the behavior, edge cases, costs, and expected results (staleFailures, genuineFailures, passing-unverified). It provides enough context for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage for the single parameter includePasses is 100%, with a clear description already. The tool description adds narrative context about passing tests but does not meaningfully enhance the parameter's meaning beyond what the schema provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Splits failures into two piles by comparing the whole `execute` chain's `dateUpdated` against each test's last run.' The title and description distinguish it as a diagnostic classifier for stale versus genuine failures, setting it apart from sibling tools like gi_validate_test or gi_inventory.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly prescribes when to use: 'Call this BEFORE diagnosing or editing any red test.' It also gives actionable guidance on where to start ('Start there, oldest first'), and warns against unsafe re-running ('Confirm what a test does before triggering it'), covering both when-to and when-not-to.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gi_validate_testA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations say readOnlyHint=false, destructiveHint=false, and openWorldHint=true, but the description adds critical context: it drives a real browser against a real URL, inlines modules, truncates at the first submit-capable step, polls for 20-100 seconds, and explains that 'passing: null' means not finished. This goes far beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though long, every paragraph earns its place: purpose, safety guard mechanics, browser/viewport context, and timing semantics. The structure is front-loaded with the core action and then progressively adds caveats, making the length justified for a tool with non-obvious execution behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description explains key return behavior (ranAs.configSource, passing:null) and runtime expectations. For a tool with nested inputs and real-world side effects, it covers the necessary context: use cases, safety guards, timing, and how to interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds context beyond the schema for browser/viewport by explaining why replication matters ('tests inherit those and a selector can resolve on desktop and fail on mobile'). It does not add new semantics for dryRun, but the schema already covers it thoroughly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Runs a test definition through on-demand execution, which executes it and discards it.' It clearly differentiates from sibling tools by framing this as pre-save validation ('check that a selector chain still resolves before editing a test').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: before editing a test to verify selectors, and before saving an authored definition. It also states a clear exclusion: 'There is no way to make this tool submit; that stays a deliberate curl.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.