Server Quality Checklist

Tool Scores

• search_directoryB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses caching behavior (24-hour default) and mentions CSG's school structure, which adds useful context. However, it lacks critical behavioral details: no information on permissions required, rate limits, error handling, or what the search returns (e.g., result format, pagination). The caching detail is helpful but insufficient for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise with two sentences, though the second sentence includes extraneous background about CSG's Forms and schools that doesn't directly aid tool selection. It's front-loaded with the core purpose, but could be tighter by focusing only on tool-relevant details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete for a 7-parameter search tool. It misses key contextual elements: what the output looks like (e.g., list of people with fields), authentication requirements hinted by the userEmail parameter, and how search logic works (e.g., partial matches). The caching and school info are helpful but don't compensate for these gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 7 parameters thoroughly. The description adds no parameter-specific information beyond what's in the schema—it doesn't explain how parameters interact (e.g., combining firstName and lastName) or provide examples. Baseline 3 is appropriate since the schema does the heavy lifting, but the description doesn't compensate with additional insights.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches the CSG directory for students, parents, and staff, providing a specific verb (search) and resource (directory). It distinguishes from siblings like 'check_authentication' or 'school_events' by focusing on directory lookup, though it doesn't explicitly differentiate from potential similar search tools that might exist.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions CSG uses Forms instead of grades and lists schools, but this is background info, not usage instructions. There's no mention of prerequisites, when not to use it, or how it relates to sibling tools like 'check_authentication' for auth needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• clear_credentialsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. 'Clear stored login credentials' implies a destructive operation that removes authentication data, but it doesn't specify scope (e.g., current session only, all users), permanence, side effects, or required permissions. For a security-sensitive tool with zero annotation coverage, this is inadequate behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded and directly communicates the core function without unnecessary elaboration. Perfectly concise for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema) and lack of annotations, the description is minimally adequate. It states what the tool does but lacks important context like behavioral details (e.g., what 'clear' entails) and usage guidelines. For a credential management tool, more completeness would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (empty schema). The description doesn't need to explain parameters, so it meets baseline expectations. No additional parameter semantics are required or provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Clear stored login credentials' clearly states the action (clear) and target (stored login credentials). It's specific and unambiguous, though it doesn't explicitly differentiate from sibling tools like logout (which might have different semantics). The purpose is immediately understandable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like logout or check_authentication. It doesn't mention prerequisites (e.g., must be logged in first) or typical use cases (e.g., security cleanup, switching users). The agent receives no contextual usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_authenticationA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the tool checks authentication status but does not disclose behavioral traits such as what 'valid' entails (e.g., token expiration, permissions), response format, error handling, or rate limits. This leaves significant gaps for an agent to understand the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and efficiently communicates the essential information, earning its place with zero waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete. It lacks details on what constitutes a 'valid' session, the return format (e.g., boolean, status object), or error cases. For an authentication-related tool, this leaves critical context missing for an agent to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'userEmail' fully documented in the schema. The description does not add any parameter-specific information beyond what the schema provides, such as default user behavior details. Baseline 3 is appropriate as the schema handles parameter semantics adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Check') and resource ('valid stored authentication session'), distinguishing it from siblings like login (authenticate), logout (end session), and clear_credentials (remove stored data). It precisely defines the tool's function without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context ('if you have a valid stored authentication session'), suggesting it should be used to verify existing authentication. However, it does not explicitly state when not to use it (e.g., vs. login for initial authentication) or name alternatives, though the sibling list provides clear options.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• school_eventsA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure and does so effectively. It reveals key behavioral traits: the default search period (3 months), the automatic extension logic (to 12 months if no events found), and caching behavior (implied by the 'refresh' parameter). This goes beyond the input schema to explain operational logic, though it could mention authentication needs more explicitly.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured in a single sentence, front-loading the core purpose and efficiently adding operational details without any wasted words. Every part of the sentence serves a clear purpose, making it easy for an agent to parse and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no annotations, no output schema), the description is adequate but incomplete. It covers the purpose and key behavior but lacks details on output format (e.g., what events look like), error handling, or authentication requirements beyond the 'userEmail' parameter. This leaves gaps for an agent to use the tool effectively in all contexts.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the input schema already documents all three parameters thoroughly. The description adds minimal value beyond the schema by implicitly referencing the 'searchMonths' default and fallback, but it doesn't provide additional semantic context for parameters like 'userEmail' or 'refresh'. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Check') and resource ('upcoming school calendar events'), making it immediately understandable. However, it doesn't explicitly differentiate this tool from potential sibling tools like 'search_directory' or 'lunch_volunteers' that might also involve school-related data, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage guidance by mentioning the default search period and fallback behavior, which helps an agent understand when to use it for event queries. However, it lacks explicit guidance on when to use this tool versus alternatives (e.g., compared to 'search_directory' for non-event data) or any prerequisites, leaving room for ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loginA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: it opens a browser for secure authentication and ensures credentials are not exposed in Claude. This covers safety and interaction method, though it lacks details on session persistence or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by a clarifying detail about browser interaction and security. Every sentence adds value without redundancy, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description adequately covers the tool's purpose and key behavior (browser-based auth, credential safety). However, it doesn't address potential outcomes like success/failure states or session management, leaving some gaps for a login tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the optional 'userEmail' parameter. The description doesn't add meaning beyond the schema, such as explaining why email is used for 'session isolation and identification'. Baseline 3 is appropriate as the schema handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Log in') and target ('CSG Veracross portal'), specifying it opens a browser for authentication. It distinguishes from siblings like 'check_authentication' by focusing on initiating login. However, it doesn't explicitly differentiate from 'set_default_user' which might relate to user setup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for authentication when needed, but doesn't explicitly state when to use this tool versus alternatives like 'check_authentication' (to verify status) or 'clear_credentials' (to remove stored data). No guidance on prerequisites or exclusions is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lunch_volunteersA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes what the tool returns ('Shows only days that need volunteers'), which is helpful. However, it doesn't mention important behavioral aspects like whether this requires authentication (though sibling tools suggest auth capabilities), rate limits, error conditions, or response format. The description adds some value but leaves significant gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise and front-loaded: a single sentence that immediately states the tool's purpose with zero wasted words. Every element (checking, resource, filtering criteria) earns its place and contributes to understanding. No unnecessary elaboration or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations and no output schema, the description provides adequate basic information about purpose and filtering. However, for a tool that presumably returns data about volunteer opportunities, the lack of output schema means the description should ideally mention what format the results come in (e.g., list of dates, structured data). The description is complete enough for basic understanding but has gaps given the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents all three parameters. The description doesn't add any parameter-specific information beyond what's in the schema (like explaining how 'date' and 'week' parameters interact or providing examples beyond the schema's examples). The baseline of 3 is appropriate when the schema does all the parameter documentation work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Check'), resource ('Lower School lunch volunteer opportunities'), and scope ('only days that need volunteers for Salad/deli and Soup positions at 10:45am-11:45am in the dining hall'). It distinguishes itself from sibling tools like 'school_events' or 'search_directory' by focusing specifically on volunteer opportunities with precise time and position details.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool: when checking for open volunteer slots in specific positions and times. However, it doesn't explicitly state when NOT to use it or mention alternatives among sibling tools (e.g., whether 'school_events' might overlap). The guidance is contextually clear but lacks explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_default_userA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that this sets a persistent default affecting 'all authentication and directory searches,' which is useful behavioral context. However, it doesn't mention security implications, permission requirements, or whether the change is reversible, leaving gaps for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that are front-loaded and waste no words. The first sentence states the purpose and benefit, while the second explains the scope of effect, with every element earning its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (mutation with security implications), no annotations, and no output schema, the description is adequate but incomplete. It covers the core purpose and usage context but lacks details on behavioral traits like error handling or return values, leaving room for improvement.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the single parameter 'userEmail' as 'Your email address to set as the default user.' The description adds no additional parameter semantics beyond this, meeting the baseline for high schema coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Set') and resource ('default user email'), explaining it configures persistent authentication settings. It distinguishes from siblings like 'login' (temporary session) and 'check_authentication' (verification) by focusing on default configuration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: 'so you don't need to provide it every time' and 'used for all authentication and directory searches.' However, it doesn't explicitly state when NOT to use it or name alternatives like 'login' for temporary sessions, missing full sibling differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• logoutA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the core behavioral trait (clearing session), but doesn't mention potential side effects (e.g., requiring re-authentication for subsequent operations), error conditions, or confirmation of success. It covers basic behavior but lacks depth for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It front-loads the core action ('Logout') and immediately specifies the target and effect, making it easy to parse and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter mutation tool with no annotations and no output schema, the description is minimally adequate. It states what the tool does but lacks details on behavioral nuances (e.g., what 'clear session' entails operationally, any return values, or error handling), leaving gaps in understanding the full impact.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters with 100% schema description coverage, so no parameter documentation is needed. The description appropriately doesn't discuss parameters, focusing instead on the tool's action, which aligns with the baseline expectation for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Logout from Veracross portal') and the effect ('clear session'), using precise verbs and specifying the target resource. It distinguishes itself from siblings like 'clear_credentials' by focusing on active session termination rather than credential management.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (when logged into the portal and needing to end a session), but doesn't explicitly state when to use this versus alternatives like 'clear_credentials' or provide exclusion scenarios. It offers clear context but lacks explicit comparative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.