Server Quality Checklist

Tool Scores

• list_bouncesC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, but it only says 'List bounced messages, filtered and paginated.' It does not disclose pagination defaults, ordering, return format, or behavior when no bounces exist. This is minimal transparency beyond the surface action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no superfluous words. It fronts the primary purpose immediately and is appropriate for a simple list operation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters, no output schema, and no annotations, the description is too thin. It omits expected return shape, pagination behavior, and any edge-case handling, which the agent would need to infer or test. The schema provides parameter details but not operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers all five parameters with descriptions (100% coverage), so the description need not repeat them. The phrase 'filtered and paginated' only generically alludes to tag/query/status and page/per_page, adding no specific meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'bounced messages', with additional qualifiers 'filtered and paginated'. It is distinct from sibling tools like list_emails because it specifically targets bounces, though it does not explicitly contrast itself with them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives such as list_emails. The description only states capabilities (filtered and paginated) but does not indicate scenarios or exclude cases, leaving the agent without clear selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_templatesB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It implies a read-only operation via the verb 'list' but does not explicitly state that it makes no changes, nor does it mention any rate limits, authentication requirements, or return format details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence of ten words, front-loaded with the action and resource. Every word adds value, with no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter, no-output-schema tool, the description adequately covers the core functionality. It lacks details on the return structure or possible pagination, but given the simplicity, the description is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description reinforces that it lists 'all' templates, implying no filtering is available, which adds meaning beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('stored message templates'), clearly indicating the tool's function. It does not explicitly differentiate from siblings like 'render_template' or 'send_email_with_template', but the verb 'list' already conveys a distinct operation, so it's clear but not fully distinguished.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It simply states what it does without mentioning use cases, exclusions, or related tools like 'render_template' or 'send_email_with_template'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dmarc_summaryB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears the burden of disclosing behavior. It does convey that the tool reads stored aggregate reports and produces a summary, which implies a read-only operation. However, it does not mention potential side effects, data freshness, authentication requirements, or what happens if no reports exist. The listed output fields add some transparency but leave gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, focused sentence that front-loads the main purpose and efficiently lists the three key output components. Every word adds value, with no repetition of the tool name or irrelevant detail. It is appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema, the description adequately conveys what the tool returns (pass rate, top sending sources, disposition totals). For a summary tool with three optional parameters, the description covers the core purpose and outputs. It could mention edge cases like empty results or date range behavior, but these are largely covered by the schema. The lack of usage context slightly reduces completeness, but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents the 'to', 'from', and 'domain' parameters. The description adds no parameter-specific meaning beyond mentioning 'stored aggregate reports', which is already implied by the schema. Per the baseline rule for high schema coverage, a 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as producing a DMARC compliance summary over stored aggregate reports, and explicitly lists the output contents (pass rate, top sending sources, disposition totals). It is distinct from sibling email tools that focus on sending, templates, or general stats. However, the description lacks a strong active verb, using a noun phrase instead of 'Calculate' or 'Get', which is a minor weakness.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any prerequisites (e.g., having stored aggregate reports) or contrast with sibling tools like get_stats. The context of DMARC summaries is implicit from the name but not explicitly articulated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It does not state that this is a read-only operation, nor does it mention any side effects, authorization requirements, rate limits, or handling of edge cases (e.g., empty results, default window limits). The mention of 'created_at time window' gives some behavioral context, but the overall transparency is limited.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is concise and front-loaded with the core purpose ('Message counters of the server'), followed by a compact list of examples and the optional filter. No redundant words or restatements of the tool name, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (two optional parameters, no output schema), the description covers the main functionality but omits details like return format, whether both parameters are required together, and handling of the time window. The absence of annotations leaves open questions about read-only status and expected output structure, making it adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage with descriptions for both parameters (to and from as ISO 8601). The description adds meaningful context by clarifying that the window applies to 'created_at', which is not present in the schema. This helps the agent understand the semantic filter, going beyond the bare parameter definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the resource (message counters of the server) and the specific metrics included (sent, pending, bounced, opens, clicks), which conveys a retrieval purpose despite using a noun phrase rather than an explicit verb. It distinguishes itself from sibling tools like send_email and list_emails by focusing on aggregate counters rather than individual messages, but it does not explicitly contrast with list_bounces or dmarc_summary which might have overlapping concepts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context: use this tool when you need aggregate message counters, optionally filtered by a created_at time window. However, it provides no explicit 'when to use this vs. alternatives' guidance, nor does it mention when not to use it (e.g., for detailed bounce lists, use list_bounces). The usage is inferred from the nature of the tool rather than explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_emailA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. 'Retrieve' indicates a read operation, and it discloses that delivery attempts are included. However, it does not mention response format, error handling, or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It clearly states the action and the key detail about delivery attempts, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description covers the core purpose and a notable behavioral detail. It lacks explicit note about response structure or errors, but given the low complexity, it is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with the id parameter fully described as a numeric message id with exclusiveMinimum 0. The description merely repeats 'by id' without adding any new semantic information beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Retrieve' with resource 'message' and identifier 'by id'. It also adds a distinguishing detail, 'including its SMTP delivery attempts', which clearly separates this from sibling tools like list_emails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a specific message id and need its details, but it does not explicitly state when to use it over alternatives or mention any exclusions. The contrast with list_emails is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• send_email_with_templateA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It does disclose one important behavior: direct fields (e.g., subject) override rendered ones. However, it does not mention failure modes, authentication requirements, or what the return value is, leaving the agent partially uninformed about side effects and outcomes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences. The first states the core purpose and mechanism (render + send), the second adds a critical precedence rule. No filler or redundancy; information density is high.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no annotations, the description should explain what the caller receives (e.g., message ID, success flag) and any prerequisites or side effects. It does neither. While parameter semantics are handled by the schema, the tool's return behavior and error conditions are completely undocumented, making it incomplete for a 10-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds meaningful context beyond the schema by explaining that template_model contains Mustache-style variables and that direct field values override rendered output. This helps the agent understand how parameters interact, justifying a score above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action (render a stored template and send it) with a clear resource (CamelMailer template) and distinguishes itself from siblings like render_template (which only renders) and send_email (which likely sends without template rendering). The verb 'render' + 'send' is concrete and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for sending templated emails but does not explicitly state when to use it over send_email or render_template. There is no mention of alternatives or exclusions, so guidance is only implicit from the wording and tool name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_emailsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of disclosing behavior. It mentions 'newest first' ordering and the available filters, which is useful. However, it does not disclose pagination behavior, return format, or whether it is a read-only operation, leaving gaps in full behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: two sentences with the main verb and object upfront, followed by filter capabilities. Every word contributes value, and it is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema and annotations, the description should explain what is returned, but it does not. It covers purpose, ordering, and filter options, which is good, but lacks details on pagination, default behavior, and return shape, making it only moderately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description enumerates several filter parameters (scope, status, tag, query, stream), but this largely mirrors the schema descriptions. It adds little beyond grouping them as filters, and omits page/per_page semantics, though those are well-described in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('List') and resource ('messages of the CamelMailer server'), and specifies ordering ('newest first'). It distinguishes itself from siblings like get_email (single message retrieval) and list_templates/list_bounces (different resource types).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description establishes the tool's purpose as listing and filtering messages, providing clear context for when to use it. However, it does not explicitly mention when not to use it or point to alternatives like get_email for individual messages, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• send_emailA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses important behavioral traits: 'Queues one message per recipient and returns their ids,' indicating asynchronous processing and per-recipient returns. It also states the need to provide a body. It does not cover auth or rate limits, but the key behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. It front-loads the action and efficiently conveys purpose, key input, and behavioral outcome.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides a solid overview of the tool's purpose, key inputs, and outcome (queued IDs). Given the rich parameter descriptions in the schema, it is reasonably complete for a high-level understanding, though it omits details like error handling or verification requirements for the 'from' field (which are in the schema).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value by highlighting html_body/text_body as the primary content options and explaining the 'to' array behavior ('one message per recipient'), which enriches understanding of how parameters relate to the tool's behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Send') and names the resource ('transactional email through CamelMailer'), clearly stating the tool's action. It also mentions key inputs ('html_body and/or text_body'). However, it does not explicitly contrast with the sibling send_email_with_template, so it lacks explicit sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the usage context (sending emails with raw HTML/plain text) and instructs to provide html_body and/or text_body. However, it does not mention when to prefer this tool over alternatives like send_email_with_template, nor does it state any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• render_templateA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full transparency burden. It explicitly discloses the key safety trait (no sending) and states the return fields. It does not discuss error behavior or side effects beyond sending, but for a preview tool the main risk is well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that communicates the primary action, the inputs, the outputs, and the crucial non-sending behavior. Every clause earns its place with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only two parameters and no output schema, but the description lists the returned fields and clarifies the no-send behavior, which is essential for safe selection. It could mention whether template_model is optional or what happens when it is omitted, but the schema already indicates it is optional, so the description is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for both parameters, so a baseline of 3 is appropriate. The description adds minimal extra meaning beyond the schema, referring to template_model as a 'variable model' and saying the template is rendered against it, but it doesn't elaborate on format, defaults, or edge cases.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Preview') and resource ('stored template'), and clearly distinguishes this from sending tools by stating it renders without sending anything. It also names the exact return outputs (subject, html_body, text_body), making the tool's purpose unmistakable and differentiating it from siblings like send_email or send_email_with_template.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'without sending anything' clearly implies this tool is for previewing before sending, providing useful context for when to use it over the send-related siblings. It does not explicitly name alternative tools or state when not to use it, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.