Server Quality Checklist

Tool Scores

• get_user_auth_methodsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states the read-only permission, the app-only limitation that leads to permission errors under azure-cli, the behavior for nonexistent accounts (returns a specific structure instead of an error), and the precise semantics of mfa_registered. It does not detail the full response structure for existing accounts, which is a minor gap, but the key behaviors are well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening statement, a code-formatted explanation of the key field, a note on nonexistent accounts, and a security/permission caveat. It is informative but not bloated; each paragraph serves a distinct purpose. It could be slightly more concise, but the structure aids readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers the MFA-checking purpose and the nonexistent-account case, but it does not specify the full response format for an existing account beyond the mfa_registered boolean. Since there is no output schema, an agent may not know if the response includes a list of methods, their types, or other fields. This is a notable gap for a tool that returns a data structure, though it may be sufficient for the primary use case of checking MFA.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the parameter name 'upn' with no description (0% coverage). The description compensates by stating 'Args: upn: The account's userPrincipalName.' This adds the meaning and domain context, making it clear what value to provide. While it doesn't elaborate on format or constraints, the parameter is simple and the explanation is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear, specific purpose: retrieving registered authentication methods for an account and determining whether MFA is actually set up. It explains the key field (mfa_registered) and distinguishes itself from sibling tools that deal with sign-in logs or audits. The verb and resource are unambiguous, and the contrast with other tools is implicit but effective.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool (to check MFA coverage) and includes a practical caveat about app-only auth under azure-cli. It references get_user's contract for nonexistent accounts, offering a form of alternative comparison. However, it does not explicitly name siblings like signin_logs or signin_failure_stats as alternatives, leaving some usage inference to the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• daily_briefA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description fully compensates for missing annotations by disclosing: synchronous execution, per-section permission degradation behavior, page budget handling, hour clamping, and the unused samples parameter. It even notes the performance consideration and potential porting path. No explicit read-only statement, but a summary tool is implicitly non-destructive and the behavior is well specified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a crisp one-line summary up front, then behavior details, then a clean Args section. Every sentence adds value, though a few extra details (e.g., roadmap reference) could be trimmed. It is appropriately sized for the tool's complexity without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity and missing output schema, the description is largely complete: it covers behavior, parameter semantics, and error degradation. It references the return shape via 'matching the shape of this fleet's other daily_brief tools' but does not explicitly enumerate return fields, which is a minor gap. However, it clearly explains the combined nature and the summary on top, sufficient for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description carries the full burden and excels: 'hours' is explained with clamp range, 'max_pages' with default and purpose, and 'samples' is explicitly flagged as reserved/unused. Each parameter gets meaningful semantics beyond the raw type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb-resource pair ('combines signin_failure_stats and directory_audits into one result') and names the exact content (sign-in failures, spray suspects, admin actions). It clearly distinguishes itself from siblings by explicitly naming the two source tools and contrasting with the job+poll pattern.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies when to use it (morning patrol, one-call summary) and differentiates from the sibling gwsadm-mcp's job+poll daily_brief. It does not explicitly list exclusions or alternatives beyond that sibling, but the purpose statement makes it obvious that this is the combined-synchronous variant.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• health_checkA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description fully carries the behavioral burden — and it succeeds admirably. It discloses read-only status, that both probes always run independently, and the exact status derivation (healthy/degraded/error), plus the output contract (always same keys, detail null on success, translated message on failure). This is thorough, non-obvious behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long (~130 words) but every sentence earns its place — the independence rationale, the derived-status logic, and the output contract are all essential behavioral details that prevent misdiagnosis. It is front-loaded with the core purpose and only moderately verbose, justified by the nuance it conveys.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no parameters and no output schema, the description carries a heavy completeness burden and mostly meets it: it explains all three status values, the constant key contract, and the translated-message behavior. The only small gap is that it references the return keys ('detail', 'status', 'graph', 'signin_probe') without enumerating the full key set a caller should expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters with an empty schema, so there is nothing for the description to compensate for. Per the baseline for 0-param tools, a 4 is appropriate; there is no parameter documentation gap to fill.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description gives a specific verb and resource ('Fleet-standard health probe: service/version/status plus two independent Graph probes') and then defines each component precisely. It distinguishes itself from the sibling data tools (get_user, signin_logs, etc.) by being the diagnostic probe rather than a data-retrieval tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that signin_probe validates 'the permission every other tool here except get_user depends on', which implicitly tells the agent this tool verifies prerequisites before relying on sibling tools. However, it never explicitly states 'use this when you need to check if deployments are healthy' or gives an explicit when-not-to-use directive, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• signin_failure_statsA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears full responsibility and delivers richly: it declares read-only status with specific permission/role requirements (AuditLog.Read.All or Reports Reader), discloses that Graph cannot filter server-side so it walks up to max_pages and aggregates client-side, and explains that capped=true means counts are a sample rather than a census. It also states the hours clamp (1–720). This exceeds what annotations would typically convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The text is front-loaded with the core purpose and output views, then layers behavioral detail and Args. It is dense but not bloated; the sentences on KeyCloak mirroring and the MCP Server/Graph gap, while adding institutional context, are slightly extraneous to actually invoking the tool. Overall well-organized with a consistent summary → behavior → args structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 0% schema coverage, no output schema, and no annotations, the description is highly complete: it names all four output views, the spray_suspects flag with its threshold, the capped sampling caveat, auth requirements, and the pagination semantics. The only minor omission is a precise shape of the returned views (exact fields per row), but the narrative gives enough for an agent to call it and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must carry parameter docs, and it does: hours ('How far back to look, clamped to [1, 720] (30 days)') and max_pages ('Page budget (default: ENTRAADM_MAX_PAGES_DEFAULT)'). It also connects max_pages to the behavioral capped flag, adding causal meaning absent from the bare schema. The minor schema/description default mismatch (null vs ENTRAADM_MAX_PAGES_DEFAULT) is benign since null triggers the env-var default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('aggregates') and resource ('failed sign-ins across the whole tenant') and enumerates exactly four output views (AADSTS error codes, failing users, applications, source IPs). It also positions itself against siblings as the 'Entra ID counterpart to the RADIUS failure patrol' and explicitly references signin_logs for error-code meaning annotations, so an agent can distinguish it from neighboring tools without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear driving use case: detecting low-and-slow password sprays that evade per-account smart lockout, capping with the concrete 5-distinct-users spray_suspects threshold, and noting that neither the official Microsoft MCP Server nor Graph offers this aggregation. It does not, however, explicitly state when NOT to use it or name the alternative tool (e.g., use signin_logs for raw per-sign-in detail), so a small exclusion gap remains.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_userA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and excels. It discloses the meaning of key fields (account_enabled, on_premises_sync_enabled, licenses_capped), the graceful degradation of sign_in_activity without the required permission, and that a nonexistent account returns {'found': false} rather than an error. It also states the read-only nature and exact UPN requirement. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though long, the description is densely packed with prerequisite triage knowledge—field semantics, permission nuances, and edge cases—and every sentence adds value. It is structured with clear paragraphs and code-formatting for fields, making it scannable. The front-loaded purpose statement ensures the agent quickly knows what the tool does.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema, the description covers the essential return fields (account_enabled, last_password_change_date_time, on_premises_sync_enabled, licenses, licenses_capped, sign_in_activity), their meanings, and expected edge behavior. It also describes permission requirements and the graceful degradation path. Nothing an agent needs to call it and interpret results is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage for the only parameter, upn, is 0%, but the description fully compensates: 'Args: upn: The account's userPrincipalName, e.g. "user@example.edu".' It also stresses the need for an exact match, not display name. This gives the agent everything needed to format the argument correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'One account's identity/lifecycle state -- the first thing to check on any triage report.' This is a specific verb+resource (get identity/lifecycle state) and explicitly positions it as the initial step, distinguishing it from sibling tools that focus on sign-in logs, auth methods, or audits. The purpose is unmistakable and actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear context: 'first thing to check' and instructs to 'stop there' if account_enabled=false. It also explains a common pattern for stale passwords. However, it never explicitly names alternative tools or states when not to use this tool (e.g., 'for auth methods, use get_user_auth_methods'). The order is implied but not contrasted with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• directory_auditsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses read-only status, required permissions (AuditLog.Read.All, Reports Reader), retention (30 days), and critically explains client-side filtering for targetResources—including the consequence for max_pages budgeting and the capped=true warning. This is thorough and actionable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than typical but every sentence earns its place—purpose, example, technical caveat, permissions, retention, and parameter docs. It's front-loaded with the central question, and the structure (overview, rationale, behavior, args) is logical and scannable. Slight verbosity in the anecdote, but not wasteful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 parameters, no output schema, and no annotations, the description covers purpose, usage, permissions, pagination behavior, retention, and parameter semantics. It also anticipates edge cases (capped=true, larger max_pages). The only minor gap is an explicit return format, but the description already implies the audit fields (initiated_by, target_resources).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the Args section explains each parameter's meaning, default, and clamp ranges (hours [1,720], top [1,500], max_pages budget). It also clarifies the semantic nuance of 'user' (actor or target). This fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the resource (directory audits), the core function (who did what to the directory), and distinguishes it from signin_logs as the operator-side counterpart. It even provides a concrete scenario (manual unblock and reset) that clarifies its unique role. This fully differentiates it from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly ties usage to a specific triage need (distinguishing 'handled by human' from 'still open'), names the alternative (signin_logs) and contrasts behavior (full-window fetch vs server-side filtering). It also gives operational advice about max_pages requirements, making when-to-use and when-not-to-use precise.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• signin_logsA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and excels. It discloses read-only permissions (AuditLog.Read.All or Reports Reader), retention limits (30 days), clamping behavior, client-side filtering, page-walking mechanics, and the meaning of capped=true. It even clarifies that low match counts with capped=true indicate a budget limit, not absence of data. This is a model of transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured, front-loading the core purpose, then providing necessary nuance about filtering and paging, then permissions, then a clean parameter list. Every sentence contributes to correct usage; there is no filler or redundancy. It is long but appropriately so for a tool with complex client-side behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (client-side filtering, pagination, permission requirements, retention window, edge cases) and the absence of an output schema, the description is remarkably complete. It covers all scenarios an agent needs to invoke the tool correctly and interpret results, including the capped flag and empty results. Nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates thoroughly. It explains each parameter in detail: user's userPrincipalName, hours clamped to [1,720], result options with default 'failure', top clamped to [1,500], and max_pages with default. It also explains the functional meaning of result (client-side filter) and the cap semantics, providing far more than the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool returns 'one user's recent sign-in events' and frames it as 'the most direct answer to why can't this person log in'. It identifies the AADSTS annotation as a key value-add, distinguishing it from sibling tools like signin_failure_stats that likely provide aggregated statistics. The purpose is specific and immediately actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage context: it is positioned as the go-to for individual login troubleshooting, and it explains important behavioral caveats like client-side filtering that require adjusting expectations. It implies when not to use it (e.g., for stats, use signin_failure_stats) and details what happens when the page budget is exhausted (capped=true). Alternatives are not explicitly named but clearly indicated by context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.