Server Quality Checklist

Tool Scores

• ai_risk_classifyA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes the evaluation process and references regulations, and notes the return is a 'structured analysis framework for the host to complete,' implying interactive use. No annotations are present, so the description carries the full burden but does not disclose all behavioral traits (e.g., side effects, auth requirements).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement, explanation of the evaluation framework, and an Args section. It is reasonably concise for the complexity, though some sentences could be tightened.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter and presence of an output schema, the description provides adequate context for the tool's purpose and input. The notion of a 'structured analysis framework' is somewhat vague, but sufficient for an agent to understand the tool's role.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by providing detailed guidance on what to include in the deployment_description parameter (system purpose, users, decisions, data, oversight). This adds significant meaning beyond the schema's basic type and title.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it classifies risk tier and applicable regulations for an AI deployment, referencing specific controls (HOC-001, EU AI Act, NIST AI RMF). Does not explicitly differentiate from sibling tools like ai_red_team and ai_safety_screen, but the distinct purpose is evident.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides guidance on when to use (evaluating deployment description against HOC-001) and what to include in the description. Lacks explicit when-not-to-use examples or alternatives, though the context of siblings might imply they are for different tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ai_safety_screenA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses that it evaluates against SAF-002 and returns a framework for the host to complete, but does not detail the evaluation process, side effects, or whether it runs actual checks. No annotations were provided to supplement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences plus a clean Args block. Efficient but could condense further. Information is well front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, description does not need to detail return format. Covers core behavior and parameters. Could elaborate on output usage or integration with other safety tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Description includes an Args section explaining both parameters (system_prompt and context) with practical examples, adding meaning beyond the schema which only provides titles and types. Schema coverage is effectively 100% via description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool screens an AI system's configuration for safety risks (SAF-001) and evaluates a system prompt against a control. Differentiates from siblings (ai_red_team, ai_risk_classify) by focusing on prompt screening versus red teaming or risk classification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for safety screening but does not explicitly specify when to use this tool versus siblings. Lacks 'when-not-to-use' instructions or conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ai_red_teamA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description must cover behavior. It states the output is a 'red team runbook' and 'structured framework', but does not disclose any side effects, permissions, or rate limits. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two paragraphs. Front-loaded with the core purpose, followed by parameter details. Every sentence adds value, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description need not detail return values. It covers purpose, output type, and parameters effectively. Could benefit from a brief example or more detail on the runbook structure, but is largely complete for a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description includes an 'Args' section that explains both parameters: system_prompt (the prompt to test) and num_test_cases (default 10, max 30). Adds significant meaning beyond the schema's bare titles and types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly specifies the verb 'Generate adversarial test cases' and the resource 'AI system prompt', with a security procedure reference (SEC-005). Distinguishes itself from siblings ai_risk_classify and ai_safety_screen by its specific red-teaming focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for testing AI prompts, but lacks explicit guidance on when to use this tool versus siblings or when not to use it. No exclusions or alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.