Server Quality Checklist

Tool Scores

• get_sessionB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves details but doesn't cover critical aspects like required permissions, error handling (e.g., if the session doesn't exist), rate limits, or whether it's a read-only operation. The mention of return values is basic and doesn't add depth beyond what the output schema likely provides.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose in the first sentence. The 'Args' and 'Returns' sections are clearly labeled and concise, with no wasted words. Every sentence earns its place by directly contributing to understanding the tool's functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no nested objects) and the presence of an output schema (which handles return value details), the description is adequate but has gaps. It covers the basic purpose and parameter semantics well, but lacks usage guidelines and behavioral transparency, which are important for a tool that likely interacts with session management in a broader system.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for the single parameter 'session_name' by specifying it as a 'Resource name' and providing an example format ('sessions/abc123'), which compensates for the 0% schema description coverage. This clarifies the parameter's purpose and expected syntax beyond the bare schema, making it highly valuable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get details') and resource ('about a specific session'), making it immediately understandable. It distinguishes this from siblings like 'list_sessions' by focusing on a single session rather than listing multiple. However, it doesn't explicitly contrast with other siblings like 'create_session' beyond the verb difference.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing session), exclusions, or comparisons to siblings like 'list_sessions' for browsing or 'create_session' for initial setup. Usage is implied by the name but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sessionsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions pagination behavior and filtering capability (active_only), which are useful behavioral traits. However, it doesn't address rate limits, authentication requirements, error conditions, or what constitutes a 'session' beyond 'coding tasks'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections for Args and Returns. Each sentence earns its place by providing essential information. It's appropriately sized for a list operation with three parameters, though the opening sentence could be more front-loaded with key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (list operation with filtering and pagination), no annotations, and the presence of an output schema (implied by the Returns section), the description provides adequate coverage. It explains parameters well and indicates return structure, though could benefit from more context about the session resource itself.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate for the schema's lack of parameter documentation. It successfully explains all three parameters: page_size (range and default), page_token (pagination purpose), and active_only (filtering behavior). The description adds meaningful context beyond what the bare schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and resource 'Jules sessions (coding tasks)', making the purpose specific and understandable. It distinguishes the resource type from siblings like 'activities' or 'sources', but doesn't explicitly differentiate from other list operations like 'list_activities' or 'list_sources' beyond the resource name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose list_sessions over get_session for individual sessions, or how it relates to list_activities or list_sources. There's no context about prerequisites, typical use cases, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_sourceA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'Get details' but does not disclose behavioral traits such as whether it requires specific permissions, rate limits, or error handling. The description is minimal and does not add context beyond the basic operation, leaving gaps in understanding how the tool behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose followed by structured 'Args' and 'Returns' sections. Every sentence adds value without redundancy, making it efficient and easy to parse for an AI agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return values), the description provides adequate context for a simple read operation. It covers the purpose, parameter semantics, and return overview, but lacks behavioral details like error cases or permissions. For a tool with one parameter and output schema, this is mostly complete, though slight improvements in transparency would enhance it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning beyond the input schema by explaining the parameter 'source_name' with an example ('e.g., "sources/github/owner/repo"'), which clarifies the expected format. Since schema description coverage is 0%, the description compensates well by providing this semantic context, though it could be more detailed about constraints or usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with 'Get details about a specific source repository,' which includes a specific verb ('Get details') and resource ('source repository'). However, it does not explicitly differentiate from sibling tools like 'list_sources' beyond implying this tool retrieves details for a single source versus listing multiple sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying it retrieves details for 'a specific source repository,' suggesting it should be used when details for a particular source are needed, as opposed to 'list_sources' for a list. However, it lacks explicit guidance on when to use this tool versus alternatives like 'get_session' or prerequisites such as authentication needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_activitiesA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that the tool returns paginated results (via page_token) and default behavior (page_size default 50), which are useful behavioral traits. However, it doesn't cover other aspects like rate limits, authentication needs, error conditions, or whether it's read-only (implied but not explicit).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement, parameter explanations, and return value note. It uses bullet-like sections (Args, Returns) for readability. Some minor verbosity exists (e.g., 'Activities represent...' could be tighter), but overall it's efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters with 0% schema coverage and an output schema present, the description does a good job explaining parameter semantics and the return structure. It covers the core functionality adequately for a list operation, though it could benefit from more behavioral context (e.g., error cases) since annotations are absent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaningful context for all three parameters: session_name is clarified as a 'Resource name' with an example format, page_size specifies range (1-100) and default, and page_token explains its purpose for pagination. This goes beyond the bare schema, though it doesn't detail validation rules beyond the range.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('activities for a session'), specifying that activities are 'individual work units within a session' showing 'conversation and actions taken by Jules'. This distinguishes it from sibling tools like list_sessions or list_sources by focusing on session-specific work history.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing session), exclusions, or comparisons with sibling tools like get_session or list_sessions, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_pull_requestA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses key behavioral traits: it creates a session that automatically creates a PR when work is complete, and mentions checking the 'outputs' field for the PR URL. However, it doesn't cover important aspects like authentication needs, rate limits, error conditions, or what happens if the session fails.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, parameters, returns) and front-loaded information. Most sentences earn their place, though the parameter explanations could be slightly more concise. The overall length is appropriate for a 4-parameter tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (session creation with PR outcome), no annotations, 0% schema coverage, but presence of an output schema, the description does reasonably well. It explains the purpose, parameters, and what to expect in returns, though could benefit from more behavioral context about the session lifecycle and error handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It provides meaningful semantics for all 4 parameters: prompt (description of changes), source (resource name of repository), branch (optional base branch), and title (optional title). It adds value beyond the bare schema by explaining what each parameter represents and providing examples for source.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool creates a session that results in a pull request, specifying it's a convenience wrapper around create_session with the specific purpose of creating PRs. It distinguishes from sibling tools like create_session by emphasizing the PR outcome and from list_sessions/get_session by being a creation tool rather than retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (to create sessions that result in PRs) and mentions it's a wrapper around create_session, implying an alternative. However, it doesn't explicitly state when NOT to use it or compare it to other sibling tools like approve_plan or send_message.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_sessionA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the creation action and mentions that Jules will wait for plan approval if required, which adds useful context. However, it does not cover other behavioral aspects such as authentication needs, rate limits, error conditions, or what happens if the session creation fails, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by clear sections for 'Args' and 'Returns'. Every sentence earns its place by providing essential information without redundancy, making it efficient and easy to parse for an AI agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a session creation tool with 5 parameters and no annotations, the description does a good job covering purpose, parameters, and return values. Since an output schema exists, the description need not detail return values extensively. However, it could improve by addressing potential errors or prerequisites, making it slightly incomplete for full contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides meaningful semantics for all parameters: 'prompt' as the coding task description, 'source' as the resource name, 'branch' as optional with default behavior, 'title' as optional with auto-generation, and 'require_plan_approval' explaining its effect. This adds significant value beyond the bare schema, though it could include more details like format examples for 'source'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Create a new Jules session') and the resource ('to work on a coding task'), distinguishing it from siblings like 'get_session' or 'list_sessions' which retrieve existing sessions rather than creating new ones. The verb+resource combination is precise and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'to work on a coding task' and mentions the 'require_plan_approval' parameter behavior, but it does not explicitly state when to use this tool versus alternatives like 'send_message' or 'create_pull_request'. It provides clear context for initiating a session but lacks explicit exclusions or comparisons to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool as a list operation with pagination support, which implies it's read-only and non-destructive. However, it doesn't cover aspects like rate limits, authentication needs, error handling, or what 'Jules can work with' entails, leaving gaps in behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by structured sections for 'Args' and 'Returns' that are concise and waste-free. Every sentence adds value, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (2 parameters, no nested objects) and the presence of an output schema (handling return values), the description is largely complete. It covers purpose, parameters, and returns adequately. However, it could benefit from more behavioral details (e.g., error cases or 'Jules' context) to fully compensate for the lack of annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaningful semantics by explaining 'page_size' as 'Number of sources to return' with a range and default, and 'page_token' as 'Token for pagination from previous response,' which clarifies usage beyond the bare schema. This adequately covers both parameters, though it could detail format constraints for the token.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('available GitHub repositories that Jules can work with'), making the purpose specific and actionable. It distinguishes this tool from siblings like 'get_source' (which retrieves a single source) and 'create_pull_request' (which performs a different action), establishing its unique role in the toolset.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by specifying 'available GitHub repositories that Jules can work with,' which helps identify when to use this tool. However, it lacks explicit guidance on when not to use it or direct alternatives (e.g., compared to 'get_source' for a single repository), missing full sibling differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• approve_planA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the state transition (AWAITING_PLAN_APPROVAL) and the action (approve), implying a mutation, but lacks details on permissions, side effects, or error handling. It adds some context but doesn't fully cover behavioral traits like what happens on failure or if approval is reversible.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured and front-loaded with the purpose, followed by usage guidelines and parameter details. Every sentence adds value without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (returns success confirmation), the description doesn't need to explain return values. It covers purpose, usage, and parameter semantics adequately for a mutation tool, but lacks details on behavioral aspects like error cases or prerequisites beyond state, leaving minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by explaining the single parameter: 'session_name: Resource name of the session (e.g., "sessions/abc123")'. This adds meaning beyond the bare schema, clarifying the format and purpose, though it doesn't detail constraints like length or validation rules.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Approve Jules's plan for a session') and the resource ('session'), distinguishing it from sibling tools like create_session or get_session. It precisely defines what the tool does without being vague or tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use this tool: 'when a session is in AWAITING_PLAN_APPROVAL state and you want Jules to proceed with its proposed plan.' This provides clear context and distinguishes it from alternatives like send_message or list_sessions, which serve different purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• send_messageA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It mentions the tool sends messages to 'active' sessions, implying a prerequisite, but doesn't disclose behavioral traits like authentication requirements, rate limits, error conditions, or what happens if the session isn't active. The description adds some context about the tool's purpose but lacks operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise. It begins with a clear purpose statement, provides usage guidelines, then documents parameters and return value in a clean format. Every sentence earns its place with no wasted words, and information is logically organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, but has output schema), the description provides good coverage. It explains purpose, usage context, parameters, and return value. The output schema existence means the description doesn't need to detail return values, and it adequately covers the essentials for this communication tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description provides meaningful semantic context for both parameters. It explains that session_name is a 'resource name' with an example format ('sessions/abc123') and that message is 'the message to send to Jules.' This adds significant value beyond the bare schema, though it doesn't cover validation rules or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('send a follow-up message') and target resource ('to an active Jules session'), distinguishing it from sibling tools like create_session or get_session. It provides a concrete verb+resource combination that leaves no ambiguity about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool: 'to provide additional context, clarify requirements, or respond to Jules when it's waiting for user feedback.' It provides clear usage context and distinguishes this from other session-related tools by focusing on ongoing interaction rather than session creation or retrieval.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.