Server Quality Checklist

Tool Scores

• count_usersA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of describing behavior, and it does so clearly: it returns a count, applies to a realm, and optionally filters by a search term. It is a read-only operation with no side effects, and the return concept ('number') is clearly stated. Some details such as exact matching semantics of the search term are not disclosed, but for a simple count tool this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the primary action and resource. Every phrase is meaningful and adds context: the count of users, the realm scope, and the optional search filtering. No filler or redundancy exists.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter, read-only count operation with full schema coverage, the description covers the essential context: what the result is and what data it operates on. Since there is no output schema, the description manages to state that a count is returned. It does not discuss edge cases such as zero matches or invalid realm names, but those are not necessary for basic invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already defines both parameters and their purpose. The description's 'matching a search term' phrase aligns with the 'search' parameter but adds little beyond what is already available. No additional meaning about format, boundaries, or interactions between realm and search is provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the exact operation: 'Return the number of users' scoped to a realm. This distinguishes count_users from sibling tools like list_users (which lists user objects) and get_user (which fetches a single user). The optional search clause adds operational specificity without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance about when to use this tool versus alternatives. The description implies that it is used when a count is needed rather than a detailed list, but it does not state that list_users should be used when user details are required. The agent must infer the use case from the tool's name and description alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_groupsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the behavioral disclosure burden. It indicates a read-style listing operation, but it doesn't describe response shape, pagination, search semantics, or realm restrictions beyond the schema. This is sufficient for a simple list operation, but not rich in behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded, information-dense sentence with no filler. Every word earns its place, and the key scope is immediately clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list operation with one required parameter and an optional search term, the description plus input schema provides enough information to invoke the tool correctly. A return value description is absent, but the absence of an output schema and the simplicity of a list operation make that a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already explains both parameters ('Realm name' and 'Optional search term'). The description adds no additional meaning beyond what the input schema provides, which matches the baseline given high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('List') and resource ('groups') with an explicit scope ('in a realm'). This clearly distinguishes it from sibling tools like list_users or list_realms, so an agent can identify what this tool operates on.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: when you need groups within a specific realm. However, it does not explicitly state when not to use it or mention alternative tools like list_users or list_realm_roles, leaving some of the selection reasoning to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_realm_rolesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description bears the full burden. 'List the realm-level roles' clarifies that this is a read-only operation and scopes the result to realm-level rather than client roles, which is useful context. It omits any note on potential pagination, authorization, or size details, but for a straightforward listing tool, this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single concise sentence that starts with the main verb 'List' and directly names the resource. There is no filler, and the phrase 'realm-level' front-loads the key differentiator, making it easy to process.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only one required parameter and no output schema, so the description sufficiently indicates what the operation does. The only possible missing piece is an explicit statement about the return value, but for a basic list operation, the information provided is enough for a capable agent to invoke it confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already defines the only parameter, 'realm,' with 100% coverage and a 'Realm name' description, so the tool description adds little beyond implying that the realm is the one containing the roles. Baseline 3 applies because the schema handles the documentation, and the description offers no extra examples or behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'realm-level roles defined in a realm.' It differentiates from siblings like list_realms and list_users by explicitly naming the entity being listed, so an agent knows exactly what this tool is for.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose makes its usage implied: when you need the roles that exist at the realm level. However, it does not explicitly state when not to use it or which sibling alternatives (e.g., get_realm, list_realms) might be more appropriate in different scenarios, leaving the routing decision to the model.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_usersA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds a useful and safety-relevant behavioral guarantee: 'Passwords and secrets are never returned by Keycloak.' This goes beyond the input schema and is especially valuable with no annotations present. However, it does not fully describe other behavioral aspects such as result formatting, sorting, or the effect of pagination defaults.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short sentences with no filler. It front-loads the core function, then adds search/pagination support and a valuable security note, all in an efficient and readable structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The input schema documents all parameters fully, and the description provides a key output-relevant guarantee that secrets are never returned. It does not explicitly describe array structure or how results are sorted, but for a simple list/search tool these are likely easily inferable or not critical. Overall, the description is adequately complete for invoking the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all four parameters already have individual explanations. The description only restates 'free-text search' and 'pagination' in prose, without adding extra meaning about parameter formats, defaults, or edge cases. A baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'List users in a realm' with a specific verb and resource, and 'Supports free-text search and pagination' further clarifies the behavior. This clearly distinguishes list_users from get_user (single user) and count_users (count only) in the sibling list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving or searching multiple users, but it does not explicitly say when to use it versus sibling tools like get_user or count_users. It lacks any stated exclusions, so it stays at implied guidance rather than explicit routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_realmA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. 'Fetch' implies a read-only retrieval, but the description does not disclose behavior on nonexistent realms, permission requirements, or whether any transformation occurs. This is acceptable for a simple getter but not richly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler or redundant restatements. 'Configuration of a single realm' precisely captures the tool's function and scope without wasting words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter read-only tool without an output schema, 'Fetch the configuration of a single realm' is largely sufficient. It could be slightly more complete by noting the alternative for listing all realms or error behavior, but those are not critical for a basic get-by-name call.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully describes the only parameter: 'realm' with type string and description 'Realm name'. The description adds no further parameter semantics beyond referencing a single realm, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Fetch') and resource ('the configuration of a single realm'), making both the action and the target unambiguous. It distinguishes itself from siblings like list_realms and get_user by emphasizing 'single' and the realm configuration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'a single realm' provides clear context for when to use this tool: when a specific realm's configuration is needed. It does not explicitly name an alternative, such as using list_realms for all realms, but the single-vs-list distinction is strongly implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_userA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the behavioral transparency burden. 'Fetch' indicates a read-only operation, but the description does not add contextual details such as not-found behavior, authentication prerequisites, or return format. It provides the minimal expected behavioral signal without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasted words. It is front-loaded with the action and resource, and directly followed by the identifying attribute, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter fetch tool without an output schema, the description and schema are enough for an agent to call it correctly. The only gap is not describing the exact return outcome when no user is found, but given the simplicity this is a minor omission.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers both parameters with 100% description coverage ('Realm name' and 'Keycloak user id (UUID)'). The description only restates 'Keycloak id' without adding semantic depth, so a baseline score of 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Fetch'), a specific resource ('a single user representation'), and a precise selector ('by their Keycloak id'). This clearly differentiates it from sibling tools like list_users or count_users, which operate over collections.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly identifies the context: when you need one user by a Keycloak id, with no ambiguity about what kind of lookup. It implies the use case without naming alternatives, but the single-user scope and key-based identification make the intended usage sufficiently clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_clientsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses a meaningful behavioral trait—client secrets are not included in the response. However, it does not address other behavioral aspects such as read-only guarantees, permission requirements, pagination, or how filtering works beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The primary purpose is front-loaded, and the second sentence adds a useful caveat without redundancy or excessive length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter read-only list tool with no output schema, the description adequately conveys the scope and an important output exclusion. It is sufficient for a straightforward client listing operation, though it could be slightly richer in explaining what the returned objects look like.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning both 'realm' and 'clientId' are already documented in the schema. The description provides no additional parameter-level insight. This matches the baseline of 3 when the schema handles the semantic load.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'List OAuth/OIDC clients configured in a realm.' It differentiates from sibling tools by focusing on clients rather than realms, users, roles, or groups. The additional note about client secrets reinforces exactly what this tool does and does not return.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: this tool is for listing OAuth/OIDC clients within a realm, which distinguishes it from sibling realm/user/group tools. However, it does not explicitly mention when not to use it or name alternatives, so it falls shy of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_realmsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the burden of explaining behavior, and it does so by mentioning credential-based visibility and allowlist filtering. The 'list' verb also communicates a read-only intent, though it does not explicitly address error cases or formatting.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the action and then embedding the filtering qualifier. Every word adds value and there is no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless list operation without an output schema, it completely conveys what will happen, who it applies to, and how the allowlist affects the result. No further detail is required for confident invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the schema leaves nothing to explain. The description's mention of visibility and allowlist behavior compensates for the lack of any structured parameters, making this baseline-4 situation fully comfortable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'List all realms'. It also includes the scope 'visible to the configured credentials' and the allowlist filter, making its purpose precise and distinct from get_realm and list_users.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives solid contextual guidance, saying this operation returns all visible realms while filtering out ones outside the allowlist. It does not explicitly contrast with get_realm or other siblings, but the 'list all' scope makes the intended use clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.