Server Quality Checklist

Tool Scores

• decide_actionC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It only states the transformation and offers no information about side effects, permissions, determinism, failure modes, or any operational implications. The agent cannot infer whether this tool has external effects or requires special conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It states the verb, the resource, and the output in a compact form, making it easy to read and grasp.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (10 parameters, 8 required, no output schema, no parameter descriptions, no annotations), this description is severely incomplete. It fails to explain the decision logic, the meaning of inputs, or the significance of the output, leaving an agent to guess at nearly everything needed to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero descriptions and the tool description provides no explanation of any of the 10 parameters. The phrase 'GATRI assurance plus policy/context' is a vague abstraction that does not map to specific fields like assuranceScore, policyPassed, impact, or sensitivity. The agent has no way to understand what each parameter means or how to construct a valid call beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific transformation: turning 'GATRI assurance plus policy/context' into a Vector Decision with the four enumerated outcomes. It names the output type explicitly and distinguishes the tool's core function, though it doesn't explicitly differentiate from sibling 'assure_action' or others.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given for when to use this tool versus alternatives. The description does not mention any conditions, prerequisites, or scenarios where this decision tool is appropriate, nor does it point to sibling tools like gatri_trust_score or risk_assessment.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• risk_assessmentC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears the full responsibility for disclosing side effects and behavior. It only says 'comprehensive risk assessment' without indicating whether this is a read-only operation, whether it returns a score, stores data, or has any side effects. The agent cannot infer safety or operational impact from this minimal text.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise, but it lacks structure and substance. It does not present key scoping or usage information up front. For a tool with six required parameters and no output schema, this is under-specified rather than appropriately compact.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity—six required parameters, no output schema, and no annotations—the description is incomplete. It fails to explain what the tool actually does with the inputs, what the result looks like, or how it relates to sibling tools like compliance_check or assure_action. The agent lacks essential context to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for all six parameters, including the 0-100 range for risk scores. The description itself adds no additional meaning or context about parameter usage, so this is the baseline score for high coverage. The agent can rely on the schema for parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs a comprehensive risk assessment for an AI agent deployment, which is a specific and informative purpose. However, it does not differentiate this tool from siblings like compliance_check or gatri_trust_score, which could overlap. The verb 'assessment' and resource 'AI agent deployment' are clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There are no conditions, exclusions, or references to sibling tools. An agent would have to infer when 'comprehensive risk assessment' is needed without any hints about scenarios or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assure_actionC
  2.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears full responsibility for behavioral disclosure. 'Evaluate' and 'Returns' imply a read-only operation with no side effects, which is a modest step. However, it does not explicitly state that it does not modify state, nor does it disclose any permission requirements, logging, or reversibility—important for an authoritative decision tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero wasted words. It front-loads the core purpose and then states the output. This is model conciseness—short and dense without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite the tool's high complexity (8 parameters, nested evidence object, no output schema), the description gives minimal context. It does not explain what a GATRI assurance record is, how to interpret the evidence fields, or any thresholds. An agent would be left guessing about the meaning of the returned record and the weight of each evidence field, making effective invocation risky.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameters. It fails to do so entirely. It mentions 'specific action' and 'context' but gives no hints about agentId, actionType, impact, sensitivity, or the large evidence object. An agent would have to rely solely on the schema, which is already present but the description adds zero value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Evaluate' and the resource 'whether an AI agent has earned authority to perform one specific action in one context', which is specific and action-oriented. It also mentions the return of a 'GATRI assurance record', adding specificity. However, it does not explicitly differentiate from siblings like risk_assessment or decide_action, so it misses the top score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives. It doesn't mention conditions, exclusions, or context that would help an agent choose it over gatri_trust_score, compliance_check, or decide_action. The description's generic phrasing offers no decision support.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compliance_checkC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only says 'evaluates,' which implies a read-only check, but does not disclose whether the operation is reversible, has side effects, requires specific permissions, or what it returns. For a compliance evaluation with 8 required parameters, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no redundant wording. It front-loads the core purpose. However, it sacrifices essential behavioral and usage details for brevity, so it is efficient but under-specified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 8 required parameters and no output schema or annotations, the description is far too sparse. It does not explain how compliance is determined, what the output looks like, how the risk categories are used, or what the boolean flags imply. An agent cannot confidently call this tool correctly based on the description alone.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all parameters are already documented in the schema. The description adds no additional parameter-level meaning or context. Per the rubric, a baseline of 3 is appropriate when the schema fully covers the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'evaluates an AI system against EU AI Act requirements.' This is clear and distinct from siblings like risk_assessment, though it does not explicitly name alternatives. The tool's purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus the sibling tools (e.g., risk_assessment, assure_action). The description does not mention prerequisites, context, or when to choose an alternative. An agent would have to infer usage from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• gatri_trust_scoreB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only says 'Compute' (implying a read-only calculation) but does not mention the return format, potential side effects, or any authentication/permission requirements. This is a minimal disclosure for a tool that likely just returns a score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences deliver the core action and the evaluated dimensions. There is zero fluff, and the purpose is front-loaded. All information is essential.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple compute tool, the description is adequate but not complete. It lacks an explicit statement of the return value (e.g., a numeric score) and does not mention any edge cases or error behavior. Given the 6 required parameters are fully documented in the schema, the main gap is the absence of output description, but this is a minor omission for such a straightforward calculation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the parameters with clear descriptions and ranges, so the baseline is 3. The description only lists the five dimension names without adding any additional meaning, such as weighting, aggregation logic, or expected values beyond the 0-100 ranges already in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Compute') and a specific resource ('GATRI trust score for an AI agent'), and names the five dimensions evaluated. It is not a tautology and is easy to distinguish from siblings like kill_switch or decide_action, even though it doesn't explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus the sibling tools (compliance_check, risk_assessment, etc.). It does not state any prerequisites, exclusions, or typical call conditions, leaving the agent to infer when a GATRI trust score is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kill_switchA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool 'immediately stops agent operations,' which is a key behavior, but it does not disclose potential consequences such as irreversibility, data loss, or interruption of ongoing tasks. For a kill switch, this is a significant omission, yet the immediate-halt nature is partially transparent. Thus a score of 3 reflects adequate but incomplete disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, with the purpose front-loaded and no wasted words. It states the nature (emergency halt) and the action (immediately stop operations) efficiently. Every sentence earns its place and the structure is optimal for quick agent comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's destructive nature (kill switch) and the absence of an output schema, the description is arguably thin. It tells agents what it does but not the consequences, the meaning of severity levels, or any prerequisites. While the schema fills in parameter details, the overall context—especially side effects—remains incomplete. A score of 3 reflects that it's minimally adequate but could be more thorough for a high-impact action.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters (agentName, reason, severity, scope) with meaningful descriptions. The tool description adds no extra parameter information, so it does not go beyond the baseline. A score of 3 is appropriate because the schema does the heavy lifting, and the description does not compensate with any additional semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Emergency halt for an AI agent' and 'immediately stop agent operations.' This is a specific verb (halt/stop) and resource (AI agent), and it is clearly distinct from sibling tools like trust score, compliance check, or risk assessment, which serve different purposes. The name reinforces the intent, making it unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates the tool is for emergency situations ('Emergency halt'), giving clear context for when to use it. However, it does not explicitly mention alternatives or when not to use it. Since sibling tools are entirely different domains (trust, compliance, risk, assurance, decision), there's no real ambiguity, but explicit exclusions are missing, so it's slightly below a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.