Server Quality Checklist

Tool Scores

• find_utility_conflictsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds meaningful context: it scans the whole portfolio, considers distance, evidence quality, and disagreement between county records and independent detection, and ranks results by consequence-weighted risk.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficient and front-loaded, with the main purpose in the first sentence. The second sentence adds useful positional context (cross-portfolio vs per-permit), and the third explains output ranking; while not as terse as a two-sentence ideal, there is no wasted wording.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a rich input schema, full schema coverage, and an output schema present, the description provides sufficient context for a complex portfolio-scanning tool. It explains the risk-based selection criteria and ranking approach, making the tool's behavior clear without needing to restate return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema coverage is 100%, so the baseline is 3. The description alludes to key semantics such as 'close to the excavation' (max_distance_ft) and 'where the county record and the independent detection disagree' (conflicts_only), but it does not systematically detail each parameter beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's purpose: scanning the entire permit portfolio for utility lines that are hazardous to dig near. It distinguishes this from per-permit lookups and emphasizes the cross-portfolio triage nature, matching the tool name and title.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly positions this as a cross-portfolio safety-triage tool and contrasts it with per-permit lookups, giving a clear when-not-to-use signal. However, it does not name specific sibling alternatives like 'get_permit' or 'portfolio_summary', so it falls short of full explicit alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_permitA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds valuable context about the response contents, including types of data returned and the presence of a verdict and verification list, going beyond annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One dense, information-rich sentence that lists all key outputs without waste. It is front-loaded with the main purpose and uses a colon to introduce specifics, making it easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter read tool with full annotations and an output schema, the description thoroughly conveys what the caller will receive. It covers the complete review and even highlights the overall verdict and verification recommendations, which are likely the most important outputs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage for permit_id, including an example and note about street names. The tool description adds no additional parameter-level meaning, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Retrieve' and the resource 'complete review for one permit case', then enumerates the specific contents (utility lines, owners, distances, grades, verdict, verification needs). This is unambiguous and distinguishes itself from sibling tools like search_permits (searching) and portfolio_summary (aggregation).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for a single permit's detailed review but does not explicitly state when to prefer this over alternatives. It lacks direct comparison to siblings or exclusion criteria, leaving the agent to infer context from scope.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recommend_verificationA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only and non-destructive behavior. The description adds valuable context: it 'Ranks by consequence-weighted risk' and ties the plan to a limited field budget, which goes beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences cover purpose, prioritization rationale, and usage modes. Every sentence contributes information without redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, prioritization approach, invocation modes, and budget-cap intent. With an output schema present and annotations clarifying safety, nothing critical is missing for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides full descriptions for both parameters (100% coverage), including the 'Omit for a portfolio-wide plan' behavior that the description repeats. No new parameter meaning is added, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States clearly it 'Produce[s] a prioritised potholing plan' with explicit scope (which lines, order, why). Distinguishes from siblings by focusing on field verification planning rather than permit search or conflict finding.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly describes two invocation modes: 'Call with a permit_id for one case, or with no arguments for a portfolio-wide plan.' However, it does not mention alternative tools or when not to use it, so it stops short of a perfect 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• portfolio_summaryA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint=false. The description adds useful behavioral context: it aggregates across the whole portfolio and includes specific metrics, which goes beyond the annotations. It does not contradict annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the verb 'Roll up', and every sentence earns its place. No filler or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter aggregate tool with a rich output schema and strong annotations, the description fully covers what it does and when to use it. No gaps are apparent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear description of group_by and its default. The description mentions city and project type, which are enum values, reinforcing their meaning but not adding substantial new information beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool rolls up the entire permit portfolio and lists concrete outputs: verdict counts, risk concentration by city and project type, corroboration, and unverified consequence exposure. This clearly distinguishes it from per-permit siblings like get_permit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Directly provides usage context: "Use this for 'how are we doing' questions rather than per-permit ones." This explicitly contrasts with per-permit tools and tells the agent when to choose this tool over alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_permitsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With readOnlyHint=true and destructiveHint=false already in annotations, the description adds scope ('pending') and return shape ('summary of each match, not the full utility line detail'). It doesn't discuss rate limits or side effects, but annotations cover the safety profile, so this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded purpose, then usage, then a pointer to get_permit. Every clause earns its place; no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a search/filter operation with no required params and an output schema, so the description doesn't need to explain return structure. It covers the main purpose, the pending-case scope, and directs to get_permit for deeper detail. This is fully sufficient for an agent to select and invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 8 parameters have descriptions in the schema (100% coverage), and the description merely restates a few filter criteria without adding new semantics. The schema already explains query, verdict, min_risk, etc., so the description adds marginal value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource ('Search and filter pending excavation permit cases') and lists concrete search dimensions (street, city, applicant, project type, permit verdict, risk level). It also distinguishes itself from the sibling get_permit by noting it returns summaries rather than full detail.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this to find permits by...' and provides an alternative path for full detail with 'call get_permit for that.' This gives clear when-to-use and a named alternative, though it doesn't contrast with the other sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.