Server Quality Checklist

Tool Scores

• list_service_accountsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the transparency burden. It discloses the operation (list), scope, and included fields, which implies a read-only behavior. However, it does not mention permissions, rate limits, or potential large result sets, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the core action and scope. Every word adds value, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no parameters and no output schema, the description covers the main output (service accounts with timestamps and rotation state) and a use case. It does not explicitly state the return format, but 'List' implies a list, making it sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool takes zero parameters, and the input schema is empty. With 0 parameters, the baseline is 4. The description correctly implies no user input is needed, so no further parameter explanation is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'List' and identifies the resource ('service accounts') with precise scope ('under every client app of the current tenant') and additional details (last-used timestamps, auto-rotation state). This clearly distinguishes it from sibling tools like list_tenant_users or query_session_anomalies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides a clear use context ('Useful for spotting dormant integrations') but does not explicitly mention alternatives or when not to use the tool. While this is not as strong as naming sibling tools, the purpose is well implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_access_reviewA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden. It reveals that the tool performs a review and returns a markdown report plus recommendations, with the recommendation types explicitly listed, implying it does not directly execute changes. It does not mention permissions or rate limits, but the read-only nature is implied by the review-and-recommendation flow.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the action and includes specific output details and recommendation types without any redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and no output schema, the description provides sufficient context: it states the process, the scope, and the output format. The enumerated recommendation types provide valuable detail for an agent to handle the response, making the description complete for this tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is trivial. The description adds meaningful context by specifying the scope ('for the current tenant'), which clarifies what the operation applies to, though it does not need to explain any inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool triggers an on-demand AI-driven access review for the current tenant, with a specific verb and scope. It also enumerates the review targets (users, service accounts, anomalies) and outputs, distinguishing it from sibling tools like query_session_anomalies or list_tenant_users, which focus on narrower tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by describing it as an on-demand review for the current tenant, suggesting it is appropriate for comprehensive access evaluations. However, it does not explicitly compare to sibling tools or state when not to use it, leaving some ambiguity about alternative tools for specific queries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tenant_usersA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the output fields and implies a read-only operation via the verb 'List', but does not explicitly state side effects, authentication needs, or pagination. For a simple list tool, this is sufficient but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences efficiently convey the action, output, and use case. Every sentence adds value, and the description is front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description specifies the output fields and use cases, which is adequate for a simple, parameterless list tool. It does not mention pagination or limits, but the lack of output schema makes this acceptable. Slightly more detail on potential volume or filtering would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema is empty with 100% coverage. Per the rubric, a baseline of 4 is appropriate since there are no parameter semantics to describe.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all users of the current tenant, specifying the exact output fields (role, active flag, passkey count, last login). This distinguishes it from siblings like list_service_accounts, which targets a different resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions it is useful for access reviews and off-boarding checks, providing clear context for when to use it. However, it does not explicitly mention alternatives or when not to use it, so it lacks exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_session_anomaliesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full transparency burden. It discloses key behaviors: the result is scoped to the calling user, limited by `limit`, and ordered newest first. The term 'recent' is vague, and no response format is given, but for a simple read-only query this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the action, and every word adds value. It efficiently covers the resource, scope, limit, and ordering without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and no output schema, the description covers the essential semantics: what is returned, for whom, with a limit and ordering. It lacks specifics about the time window for 'recent' and the exact response structure, but given the simplicity, it is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description is the only source explaining the `limit` parameter. It clearly states that the tool returns 'up to `limit` rows', giving explicit meaning to the parameter beyond its type and bounds.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Return'), identifies the resource as 'session anomaly events', enumerates the event types, and scopes it to 'the calling user'. It is easily distinguishable from sibling tools like generate_policy or list_tenant_users.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: to fetch recent session anomalies for the current user. It implies a security monitoring use case, but it does not explicitly mention exclusions or alternatives among the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_policyA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states the read-only nature ('the compiled policy is NOT applied'), lists the return fields (policy, warnings, cost, tokens, latency), and points to the separate enforcement call. This goes well beyond minimal and fully informs the agent of side effects and results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and every clause earns its place. It packs in the action, output, read-only caveat, and the next-step endpoint without any wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter compile tool with no output schema or annotations, the description is remarkably complete. It covers what it does, what it returns, that it's non-mutating, and how to actually enforce the result. No important aspect is left undisclosed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only type and length constraints for the sole parameter 'intent', with 0% descriptive coverage. The description compensates fully by explaining what 'intent' means (natural-language RBAC intent) and giving concrete examples, making the parameter's semantics clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Compile') and resource ('natural-language RBAC intent' into 'Sytadel PolicySet JSON'). This uniquely distinguishes it from sibling tools like query_session_anomalies or run_access_review, which serve wholly different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for use: compile intent into policy but not enforce it. It explicitly names the separate call (PUT /policies/:tenantId) needed to apply the policy, effectively explaining when to use this tool versus the enforcement step. However, it doesn't explicitly contrast with sibling tools, though those are unrelated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.