Server Quality Checklist

Tool Scores

• ingest-contractA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint=false, so the write behavior is expected. The description adds context about what happens during ingestion (id assignment, extraction of clauses, deadline, obligations, portfolio addition). No side effects like duplicate handling are mentioned, but core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the primary action. Every word earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with two parameters and no output schema, the description covers the overall process adequately. It doesn't specify the return value, but the main behavior and outcome are described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema descriptions already cover both parameters fully (contractText and title). The description reinforces 'raw text' but doesn't add new parameter-specific meaning beyond what the schema provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool stores a contract from raw text and details the specific actions (assigns unique id, extracts key clauses, deadline, obligations, adds to portfolio). This distinguishes it from sibling tools like review-portfolio and set-company-profile.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the use case: when you have raw contract text and want to store/process it. It doesn't explicitly mention alternatives or when not to use, but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set-company-profileA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (idempotent, non-destructive, not read-only), the description adds the session-scoping detail ('for this session') and the downstream dependency on other tools. This clarifies that the tool is a stateful setter, though it does not mention the effect on previously stored profile values.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The first sentence states the action and parameters, and the second explains why the profile matters to other tools, making every word valuable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential aspects for a state-setting tool: what it stores, the session scope, and its impact on sibling tools. Combined with the idempotency and non-destructive annotations, the agent has sufficient context to invoke it correctly without needing an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% coverage with detailed descriptions for all four parameters, including an enum for riskTolerance. The description merely lists the parameter names (using 'headcount' for companySize) without adding additional meaning beyond the schema, so it stays at the baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Store' and the resource 'company context' with specific fields (industry, headcount, jurisdiction, risk tolerance). It also explicitly notes that every other Contract Sentinel tool reads this profile, which distinguishes it from the sibling tools and establishes its role as a session-level setup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool should be used before other Contract Sentinel tools, since those tools read the profile when scoring contracts. It provides clear context for when to use it, but lacks explicit 'when not to use' guidance or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• review-portfolioA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, so the description doesn't need to cover safety. It adds valuable context by specifying the output format (cards with risk score, classification, exact clause text, recommended action) and that it renders a widget, which goes beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the primary purpose, and includes only necessary details. There is no redundant repetition of schema or annotations, and every clause adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description must explain the return value, and it does so thoroughly (card contents and filter behavior). The tool is simple (one optional parameter), and the description covers the essential behavior, making it complete for an agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The filter parameter is fully documented in the schema (enum and description), and the description reinforces its meaning by listing the filter options ('safe', 'danger', 'needs_attention') and noting the default behavior ('all'). This adds interpretive light beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the verb 'Return' and resource 'contract board', and enumerates the exact card contents (risk score, classification, clause text, recommended action). This distinguishes it from sibling tools (set, ingest, run) which are clearly different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (to review the contract portfolio) and mentions optional filtering, but does not explicitly name alternatives or state when not to use it. The sibling tool names make the distinction obvious, so this is a minor gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run-sentinel-cycleA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully discloses the mutation behavior: it 'act[s] by setting status to needs_attention and recording a recommended action'. It also explains the decision triggers (deadline close, high risk, never reviewed) and mentions rendering the widget. Annotations are consistent (readOnlyHint false, destructiveHint false), and the description adds context beyond annotations, such as the specific recorded actions and evidence gathering.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: first describes the core loop, second provides scheduling/usage context, third mentions the UI output. No filler or repetition; front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with a moderately complex loop, two optional parameters, and no output schema, the description covers the full workflow, side effects, intended usage, and a key UI outcome. It does not need to explain return values since no output schema exists, and the schema covers parameters. Very complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters (dryRun and onlyContractId) with clear descriptions. The description does not add parameter-specific meaning, but baseline 3 is appropriate since the schema carries the semantic load. No gap to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Run the autonomous Contract Sentinel agent loop') and resource ('entire tracked portfolio'), and details the workflow (perceive, score, decide, act). This clearly distinguishes it from sibling tools like review-portfolio (which likely only reviews) and ingest-contract (which likely ingests).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states it is 'designed to run on a schedule in production; runnable on demand here', giving clear context for when to use it. It does not explicitly name alternatives or exclusions, but the 'entire tracked portfolio' scope and 'on demand' note imply usage patterns. A brief mention of when not to use (e.g., prefer review-portfolio for quick review) would make it 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.