Server Quality Checklist

Tool Scores

• pk_case_searchA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint. The description adds value by noting that the search is 'server-side' and operates on a fixed dataset of 1,414 judgments, which provides additional behavioral context beyond what annotations state. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 14 words, front-loaded with the key action 'Full-text search'. Every word is purposeful without redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 parameters, full schema coverage, annotations, and output schema exists), the description sufficiently covers what the tool does, its scope, and server-side behavior. It does not need to explain return values due to output schema, and the count of judgments provides useful context. Minor gap: no mention of sorting or pagination, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for both parameters: query (free-text keywords with example) and limit (max results, default 10, range 1-20). The description adds only general context ('Full-text search over 1,414 judgments') but does not enhance parameter meaning beyond schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'search' and the resource 'Supreme Court of Pakistan judgments', and it distinguishes from sibling tools like pk_get_decision (retrieving a specific decision) and pk_search_laws (searching laws). The scope is specific: 'over 1,414 judgments'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides the context of use (searching judgments) but does not explicitly state when to use versus alternatives or when not to use. The sibling tools give some implicit differentiation, but no direct guidance on exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pk_search_lawsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, etc. Description adds behavioral context: searches 967 statutes, uses title-weighted ranking. This goes beyond annotations by revealing ranking logic, though doesn't detail output format or pagination.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence (10 words) with no waste. Front-loaded with key purpose: 'Keyword search over Pakistan's 967 federal statutes'.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With output schema present, description needn't detail return values. Mentions statute count and ranking method – adequate for search tool, but could add brief guidance on query syntax or ranking implications.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of parameters with descriptions. Tool description reinforces query as free-text keywords and adds ranking context, exceeding baseline value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states 'Keyword search over Pakistan's 967 federal statutes (title-weighted ranking)' – specific verb (keyword search), resource (Pakistan's federal statutes), and distinguishing feature (title-weighted ranking). Clearly differentiates from siblings like pk_case_search and pk_get_law.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when-to-use or when-not-to-use instructions. Sibling tool names provide contextual hints (e.g., pk_get_law for direct access, pk_case_search for cases), but description does not directly guide selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pk_get_decisionA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is clear. The description adds that it fetches the 'full text', which is a behavioral detail not fully covered by annotations. There is no contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single, front-loaded sentence with no wasted words. Efficiently conveys the tool's action, resource, and input options.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple fetch tool with complete schema annotations (100% coverage) and an output schema, the description is sufficiently complete. It clarifies the input options and the result ('full text'), meeting all needs for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed descriptions for both parameters. The description only summarizes the input methods ('by row index or registry citation'), adding marginal value beyond the schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses specific verb 'Fetch' and clearly identifies the resource as 'one Supreme Court judgment's full text'. It distinguishes from siblings by specifying the input methods (row index or registry citation), making the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states the tool is used to fetch a judgment by row index or citation, implying it is the appropriate tool when you have one of these identifiers. It does not explicitly exclude scenarios or name alternatives, but the context from sibling tools (e.g., pk_case_search for searching) provides clarity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pk_get_lawA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint, idempotentHint), the description discloses that large statutes are truncated, which is a behavioral trait affecting the result. This adds value beyond the safety profile annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two concise sentences, no wasted words. The main action is front-loaded, and the truncation caveat is placed second.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, clear purpose, output schema present), the description sufficiently covers the essential information: what it does, how to get the input, and a critical limitation (truncation). No further details are necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The parameter 'law_id' is described in the schema as 'the corpus file name', but the description adds concrete context: it comes from pk_search_laws and provides an example format. With 100% schema coverage, this extra context elevates clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Fetch the full text of one federal statute.' It also specifies the scope (one statute) and a key constraint (truncation for large statutes). This differentiates it from siblings like pk_case_search and pk_get_decision.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly guides usage by mentioning truncation, warning agents that large statutes may not be fully returned. It also references pk_search_laws for obtaining the law_id, suggesting a preceding search step. However, explicit when-to-use or when-not-to-use compared to siblings is absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pk_verify_citationsA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, destructiveHint=false. Description adds significant behavioral context: extracts specific citation types, checks against corpus, performs trigram content matching for parenthetical descriptions, and provides range hints for non-existent sections. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Five well-structured sentences, each adding unique value. Front-loaded with main purpose. No extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple citation types, trigram matching, range hints), the description is complete. Output schema exists (as per context signals), so return values need not be explained. All essential behavioral aspects are covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for both parameters. Description adds value beyond schema by explaining what types of citations are expected in 'text' and the default cap for 'max_citations', plus content-checking behavior. Provides extra meaning not present in schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool verifies Pakistani legal citations, lists specific citation types (statute sections, act coordinates, Constitution articles, Supreme Court registry citations) and explains the verification process. Distinguishes from sibling tools which are search-oriented.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies use for verifying citations in legal texts like LLM answers, memos, or pleadings, but does not explicitly state when to use vs alternatives or when not to use. Sibling tools are for searching, not verification, so guidance is implicit but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.