Server Quality Checklist

Tool Scores

• get-managerB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must bear the full burden of behavioral disclosure. It adds the useful fact that the result is null at the top of the hierarchy and that only 'core attributes' are returned. However, it does not disclose other potential null cases (e.g., invalid userId), whether the operation is read-only, error behavior, or data sensitivity. Some behavioral context is provided, but significant gaps remain.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that front-loads the key purpose and includes the important null case. Every word earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple one-parameter tool, but it has no output schema and no annotations, so the description must fully explain behavior and output. It mentions 'core attributes' without specifying them and ignores how errors are handled or what happens for a non-existent user. The description is insufficient for an agent to fully understand the tool's behavior and return format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one required parameter, userId, with no description. The description says 'a user's manager' but never explicitly ties this to the userId parameter, nor does it explain the parameter's format, constraints, or meaning beyond the obvious. With schema description coverage at 0%, the description should compensate but does not, leaving the parameter meaning largely implicit from the tool name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description defines the resource (a user's manager) and indicates the return value includes core attributes and null at the top of the chain. The verb 'get' is present in the tool name, making the action clear. It is distinct from sibling tools like polyarchy-expand or get-photo, though it lacks an explicit verb in the description itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any exclusions, prerequisites, or alternative tools like polyarchy-search or visualize-identity. The context is purely informational about the return value, not about selecting the right tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-photoA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It transparently states the return format (data URI, 48x48) and the null behavior when no photo exists. It does not address error cases or side effects, but as a read-only getter, this is reasonably complete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise phrase that conveys the essential information without filler. It is front-loaded with the core purpose and output characteristics, adhering to the principle that every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter getter with no output schema, the description covers the key aspects: return type, size, and null case. It could benefit from mentioning the parameter explicitly, but given the tool's low complexity, the description is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero description coverage for the single parameter 'userId'. The description only says 'for a user', which hints at the parameter's purpose but does not explain its format, constraints, or relationship to other user identifiers. The description adds minimal value beyond the schema's property name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool's function: retrieving a user's photo as a data URI with a fixed size (48x48), and notes the possibility of null. This is a specific resource (photo) with a clear output format, distinguishing it from sibling tools like visualize-identity or get-manager.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description implies a simple photo retrieval, but there is no explicit context, prerequisite, or comparison with sibling tools such as visualize-identity, which could be a potential alternative for user identity information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• polyarchy-searchA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does disclose a useful behavioral trait—returning only the top 15 matches with core attributes—but it omits context such as required authentication, case sensitivity, partial matching, or what 'core attributes' includes. This is a minimum viable disclosure for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the verb and resource, with a parenthetical for constraints. Every word adds value and there is no repetition of schema information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a low-complexity tool with one parameter, the description covers purpose, parameter semantics, and result limits. The main gap is lack of authentication context, especially with a set-access-token sibling tool, but the absence of an output schema is acceptable since the parenthetical describes the shape.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It defines the single parameter 'term' as a name or UPN, adding essential meaning not present in the input schema. It could go further by explaining matching behavior, but the basic semantic is covered.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Search Entra ID users by name or UPN') and a clear resource scope, distinguishing it from sibling tools like get-photo or get-manager. The result constraint ('top 15 matches with core attributes') further clarifies what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use the tool: when searching Entra ID users by name or UPN. However, it does not explicitly mention alternatives or when not to use this tool, so it stops short of the 'explicit when/when-not' bar.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set-access-tokenA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the token is provided/refreshed and explains the expiresOn default behavior (ISO 8601, defaults to exp claim or 1 hour). This is meaningful beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the key action ('Provide/refresh'), and includes a parenthetical example without waste. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no output schema), the description covers purpose, condition, and param semantics. It does not mention return values or error behavior, but those are less critical for a token-setting tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains expiresOn semantics in detail (ISO 8601, default behavior). The accessToken parameter is not explicitly described, but its role as the Microsoft Graph bearer token is implied. This partial compensation warrants a mid-range score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to provide or refresh a Microsoft Graph bearer token. It specifies the resource (bearer token) and the action (provide/refresh), and distinguishes itself from sibling tools like get-auth-status and visualize-identity by focusing on setting the token server-side.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context: use when USE_CLIENT_TOKEN=true, with an example (EntraPulse SSO). It does not explicitly mention when not to use it or alternatives, but the condition is stated clearly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-auth-statusA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden of behavioral disclosure. It goes beyond a simple field list by noting that token renewal is silent, which is a useful behavioral trait. However, it does not explicitly state whether the operation is side-effect-free or describe failure modes, leaving some room for interpretation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, information-dense sentence that front-loads the returned fields and ends with a specific use case. Every phrase earns its place, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and no output schema, the description is largely complete by enumerating the returned fields and the diagnostic scenario. It could add detail on data types or the not-authenticated case, but for diagnosing auth failures it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and an empty input schema, so the baseline of 4 applies. The description correctly needs to explain no parameters, and it adds no unrelated parameter information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies what the tool returns—auth mode, signed-in account, client app id, decoded token scopes/roles, and token expiry—and ties it to a specific diagnostic purpose. Although it lacks an explicit verb, the tool name 'get-auth-status' plus the detailed field list make the function unmistakable and distinguish it from siblings like set-access-token.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the intended use case: 'use to diagnose 401/403s and missing consent.' It provides clear context for when to use the tool, though it does not mention when not to use it or name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• polyarchy-expandA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses the key behavioral trait that the tool does not update an open polyarchy canvas and returns the delta only to the caller. It also explains behavior per nodeType. It does not mention rate limits or auth, but for a fetch operation the critical side-effect nuance is covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences, each earning its place. It front-loads the core purpose, then breaks down behavior by node type, and ends with the important side-effect note. It is information-dense without being verbose, and well-organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (6 params, no output schema, no annotations), the description covers the core behaviors and usage patterns. It explains the node types, dimensions, and the delta concept. It could mention output format or error cases, but for a fetch tool this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67%, so the description compensates by explaining how parameters combine: 'For nodeType attribute pass attr+value to load the whole cohort' and 'Dimensions for a user node: org, groups, access, attributes.' This adds meaning to nodeType and dimension that the schema doesn't fully describe. However, it doesn't elaborate on every parameter, so not a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb+resource: 'Fetch one node's relationships from Microsoft Graph as a nodes/edges delta.' It clearly distinguishes itself from siblings by describing node types and dimensions, and explicitly notes it does not update a canvas, setting it apart from visualization tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use it: it explains that for group/role nodes it returns members, and for attribute nodes you pass attr+value. It also clarifies that the delta is returned to the caller only and doesn't update the canvas. However, it does not explicitly name alternative tools or state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• visualize-identityA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses several important behaviors: no-argument behavior (signed-in user in delegated mode), mode requirements (app-only needs search or userId), and the disambiguation behavior when search matches multiple people (returns candidates instead of opening). It also sets expectations that no further action is needed after opening. While it doesn't cover error cases like invalid userId, the described behaviors go well beyond a generic 'opens a visualization' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph of about 100 words, front-loaded with the primary purpose and trigger phrase. Each sentence adds necessary detail: use cases, mode constraints, disambiguation, and post-open behavior. It is slightly longer than ideal but remains efficient given the tool's complexity. Structuring as bullets could improve readability but is not required.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of annotations and output schema, the description covers the essential context: what the tool does, how to invoke it (with/without args), mode requirements, ambiguous search behavior, and follow-up expectations. It could mention what happens when a supplied userId does not exist, but for a UI-opening tool, the provided context is sufficiently complete for an agent to decide when and how to call it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing baseline meaning for both parameters. The description adds significant semantic context beyond the schema by explaining the no-argument case, when each parameter is appropriate (delegated vs app-only), and how ambiguous search results are handled. This helps the agent understand the practical usage of the parameters, not just their literal meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb and resource: 'Open the interactive Identity Polyarchy — a live relationship graph over Microsoft Entra ID'. It also distinguishes itself from siblings by noting that it opens the visualization, while polyarchy-search is for finding names and polyarchy-expand/report handle other actions. The trigger phrases ('open the polyarchy', 'show me the polyarchy around <name>') further clarify scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: 'Call when the user asks to visualize, explore or map identity relationships'. It also gives concrete alternatives: 'use polyarchy-search first for names you suspect are common' and explains when to pass search vs userId. The delegated vs app-only mode distinction offers clear contextual prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• polyarchy-reportA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden. It discloses default behavior (dimensions default to 'all'), the no-argument behavior (reports on signed-in user, delegated modes only), and the ambiguous-name fallback (returns candidates, re-call with userId). It also details what each dimension returns (e.g., full manager chain, group types, assigned vs dynamic membership). However, it does not mention error handling, pagination, or permission requirements, which would push it to a 5.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the first sentence states the core purpose and constraints. The subsequent sentences efficiently deliver usage context, dimension details, and special-case behavior without redundancy. Every sentence earns its place, and the structure is easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of an output schema and annotations, the description does a strong job of covering all necessary context: purpose, default behavior, dimension semantics, and special cases. It doesn't describe the exact JSON structure or include examples, which would be needed for a perfect score, but for a reporting tool the level of detail is sufficient for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  While the schema provides basic descriptions (100% coverage), the tool description adds substantial meaning beyond the schema. It elaborates on the dimensions enum values, explaining exactly what 'org' includes (full manager chain to the root + direct reports) and what 'groups' includes (group types, assigned vs dynamic). It also clarifies that 'search' may return ambiguous candidates and that 'userId' is the exact identifier for a re-call, which is not in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Structured JSON report of one user's identity relationships' — clearly stating the tool's function. It also distinguishes itself from the visual sibling by noting 'no UI needed, works headless' and enumerates the report dimensions (org, groups, roles, etc.), which makes the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use the tool: 'Use when the user wants analysis, a summary or the underlying data — after exploring the polyarchy visually, or instead of opening it.' This provides a clear use case and contrasts with the visual exploration alternative. Additionally, it explains when to use the userId parameter ('Ambiguous names return candidates — re-call with userId') and the delegated-modes-only restriction for no person argument.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.