Server Quality Checklist

Tool Scores

• memoryA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the burden of disclosing behavior. It explains the output fields and gives interpretive guidance, but does not mention whether the tool is read-only, if it triggers side effects, or if it requires authentication. For a listing tool, this is adequate but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loads the core purpose, and each clause adds value: the first lists fields, the second provides actionable guidance. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema) and no annotations, the description is complete enough. It explains what the output contains and how to interpret it for maintenance actions. It could mention if it only returns a summary or full details, but the field list implies enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema fully describes the single parameter 'repo' with a description, so schema coverage is 100%. The description does not add additional semantic meaning beyond what the schema provides, and since parameter count is only 1, the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists repo review-memory entries and specifies the exact fields (fingerprint, grade, reason, created, lastApplied). It distinctly differentiates from siblings like 'remember' (likely adds entries) and 'context' (likely broader context retrieval) by focusing on listing memory entries with a specific purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides usage guidance by describing when to use the lastApplied field to identify dead pricing and repeated entries for promotion, but does not explicitly exclude alternatives or mention when not to use it. Sibling tools are not directly referenced, but the workflow context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• contextA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description must disclose behavior. It clearly explains the data it returns (cyclomatic complexity, git churn, last-touched date) and its multi-language coverage via lizard. It doesn't state side effects, but the read-only nature is strongly implied by the context (profiling for review).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the core value proposition and ending with actionable guidance. No redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description sufficiently explains the tool's purpose and output for an agent to use it appropriately. It covers what metrics are returned and how to interpret them (high complexity + high churn → deep review). No formal output schema exists, but the described behavior is complete enough for decision-making.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters (repo and files) are fully described in the schema (100% coverage). The description adds no additional parameter-specific guidance beyond the schema, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states this tool provides prioritization context (complexity, churn, last-touched) for reviewing a change, explicitly distinguishing it from findings. It is specific about what it computes and names the tool's purpose precisely.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit guidance: 'use this to decide where to dig before reading code' and excludes findings ('NOT findings'). It doesn't name sibling tools as alternatives, but the context for when to use is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rememberA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on the burden of behavioral disclosure. It explains that only drops are stored, that anchoring ties the verdict to a specific line, and that the memory entry dies when that line changes. It also clarifies grade semantics, adding value beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core action and then explains grades and anchoring. It is concise, with each sentence contributing meaningful guidance—no fluff. It could be slightly tighter but is well-organized and readable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a write operation with 7 parameters and no output schema, the description covers the essential behavioral context: what is stored, when to use anchors, and the side effect of anchor invalidation. It does not describe the return value, which is acceptable for a write operation without an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is high (86%), so the baseline is 3. The description adds semantic detail to the anchor parameters (pin to source line, memory dies on change) and clarifies grade meanings, going beyond the schema's lean field descriptions. This enriches the agent's understanding of parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: persisting graded verdicts to a review memory file. It specifies the verb (persist), the resource (.leveret/memory.jsonl), and the effect (prevents re-surfacing ungraded findings). It also distinguishes from siblings by noting only drops are stored and actionable findings are reported, not remembered.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides usage context such as when to use anchors (instance verdict) vs omit (class-wide verdict) and explains that only drops are stored, implying when not to use it for actionable findings. It does not explicitly name alternative tools (like 'memory'), but the context makes the appropriate scenario clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ast_searchA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses key behavior: syntax-aware AST matching via ast-grep, the use of metavariables, and the return format. It could add details about error handling or read-only guarantees, but for a search tool the disclosed behavior and output are useful and unambiguous.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: first states the core capability, second gives usage guidance, third describes the return value. No repetition of schema details; front-loaded with the most important information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 4 params and no output schema, the description is largely complete: it covers what the tool does, when to use it, the mechanism, and the return format. Minor gaps include not addressing invalid patterns or language support nuances, but these are not critical for a search-oriented task.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds meaningful semantics for the pattern parameter by explaining metavariables ($X, $$$ARGS) and providing an example, complementing the schema's terse 'ast-grep pattern' description. Other parameters are adequately covered by the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Structural code search via ast-grep' with specific mechanism ('match a syntax-aware pattern'), resource (code), and output ('Returns file/line/matched text'). It also distinguishes itself from text grep by emphasizing syntax-aware matching with metavariables.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use it: "Use for 'every call site shaped like this' questions a text grep gets wrong," providing clear context. It contrasts with text-based grep, which implies an alternative, though it does not name a specific sibling tool or explicit when-not-to-use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scanA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that findings are review leads, not verdicts, and must be validated against current code. It also explains that suppressions are tallied with reasons and never silent, which is valuable behavioral context. However, it does not mention side effects, permissions, or performance characteristics, but the core behavioral traits are well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded: it starts with the core action, lists engines, explains the two modes, and ends with profile behavior. Every sentence adds value, and it is concise without being terse. The length is appropriate for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple engines, two modes, profile handling) and the absence of an output schema, the description does a good job covering the essential context. It explains the change-set scope, the profile mechanism, and the nature of findings. It could mention what the normalized findings look like or how to interpret them, but the description already states they are review leads, which is sufficient for an agent to proceed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage with descriptions for all five parameters. The description adds context about the base mode (scans base...HEAD changed files, secrets over base..HEAD commits) and the profilePath override, which enriches the schema. However, since the schema is already thorough, the description's added value is moderate, so a baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs multiple static-analysis engines over a change set and returns normalized findings. It specifies the engines (semgrep, gitleaks, shellcheck, ruff, actionlint) and the two modes (base ref or explicit files), distinguishing it from sibling tools like ast_search or context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly explains when to use the tool (scanning a change set) and provides two usage modes with clear instructions: give either a base git ref or an explicit files list. It also mentions the optional profilePath and how profiles scope engines, giving the agent actionable guidance on invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.