Server Quality Checklist

Tool Scores

• okta_check_sessionA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Provides specific behavioral details beyond annotations: launches headless browser, uses cookies, checks for redirect. Annotations already indicate safe read operation; description enriches with mechanism.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise yet comprehensive: states purpose, explains mechanism, lists arguments with details, specifies return format. No superfluous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, mechanism, parameters, and return format. Annotations provide safety profile. Could mention prerequisites (must have stored session from login) but implied. Good completeness for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters described with constraints and usage context. URL: must match login URL; timeout: range and default. Schema had no descriptions, so description fully compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States clear action: verify session validity for a URL. Distinguishes from sibling tools like login, list, delete, get cookies by focusing on validation. Includes specific method (headless browser, checking for redirect).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage after login to verify session, but does not explicitly state when to use this tool over alternatives like okta_list_sessions. No exclusions or when-not guidance provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• okta_list_sessionsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false. The description adds valuable behavioral context beyond annotations, notably that the tool does NOT verify session validity. This helps the agent understand a key limitation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise: two short paragraphs and a return format example. It front-loads the purpose, explains the limitation in one sentence, and provides the output format. Every sentence provides necessary information without repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and an output schema described inline, the description covers the tool's behavior, return format, and limitation about session validity. It does not mention error handling or rate limits, but for a simple read-only list tool with good annotations, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, so the input schema fully covers them (100% coverage). The description is not required to add parameter details, achieving the baseline of 4 for zero-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all stored authentication sessions' with a specific verb and resource. It distinguishes from sibling tools by explicitly stating it does NOT verify session validity and directs users to okta_check_session for that purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear context for when to use the tool (listing all sessions) and an explicit alternative for verification (okta_check_session). It does not cover when to use versus other siblings like okta_login or okta_delete_session, but the purpose is distinct enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• okta_delete_sessionA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate destructiveHint=true. The description adds valuable context beyond annotations: it specifies that the tool removes 'the session file and metadata from ~/.okta-auth/sessions/' and that re-authentication is required. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: a single-sentence summary followed by a brief explanation and structured Args/Returns. It is front-loaded with the main action and every sentence adds value without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one required parameter, no enums, output schema exists) and the richness of annotations, the description covers all necessary information: what it does, what happens (file removal, re-auth needed), and the return format. No gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage for the single parameter 'url'. The description adds meaning: 'URL whose stored session should be deleted,' clarifying the parameter's purpose beyond its name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Delete a stored session for a given URL,' specifying the verb (delete) and resource (stored session). This distinguishes it from sibling tools like okta_check_session, okta_login, okta_list_sessions, and okta_get_cookies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the consequence of use: 'The agent will need to re-authenticate to access the service again,' providing clear context for when this tool is appropriate. It does not explicitly mention when not to use or list alternatives, but the context is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• okta_get_cookiesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, non-destructive behavior. The description adds context about returning raw cookie data from a Playwright session, and specifies the return format including error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a one-line summary, usage context, and clearly separated Args and Returns sections. No wasted sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage, parameter details, return format, and error responses. Given the tool's simplicity (2 params, output schema present), it provides complete guidance for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema description coverage, the description's Args section fully explains both parameters: url (the URL whose cookies to retrieve) and domain_filter (optional substring filter), providing clear semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Retrieve cookies from a stored session.' It uses a specific verb and resource, and the tool's purpose is distinct from siblings like okta_check_session or okta_login.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a use case ('making authenticated HTTP requests without launching a browser') and explains the tool's utility, but does not explicitly contrast with siblings or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• okta_loginA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (idempotent, non-destructive), the description reveals that the tool launches a headless browser, performs TOTP MFA if configured, persists sessions to ~/.okta-auth/sessions/, and details credential fallback logic. This provides substantial behavioral context that aids agent decision-making.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: opening purpose sentence, brief behavioral overview, standalone credential resolution paragraph, clearly labeled Args section, and Returns specification. Every sentence contributes value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, 1 required, credential fallback, MFA handling, session persistence), the description covers the overall flow, parameter details, return format, and credential resolution. The output schema is described in the Returns section, making it self-contained. Minor details like error handling could be inferred from the 'message' field in the return.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by documenting all 6 parameters in an Args section. It explains fallback order for credentials (explicit → env var → keyring), the purpose of 'headed' for debugging, and timeout range/default. This adds critical meaning beyond the schema's bare names and types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states the tool's function: 'Authenticate to a website via Okta SSO and persist the session.' This specific verb+resource combination immediately distinguishes it from sibling tools (check, list, delete, get_cookies) which deal with session inspection and cleanup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool (to perform Okta login including MFA) and gives detailed information about credential resolution order and headed mode for debugging. However, it does not explicitly exclude alternatives or mention when not to use this tool, though the context of sibling tools makes this largely inferable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.