Server Quality Checklist

Tool Scores

• search_release_notesA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses the search scope (release notes, patches, etc.) and the cross-version capability, but does not mention result format, pagination, or any limitations, leaving some behavioral aspects unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, succinct sentence that starts with the action verb 'Search' and enumerates the specific resource types. No filler or redundant phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple search tool with three parameters fully described in the schema, the description conveys the fundamental purpose effectively. However, without an output schema, it could have described the return value shape, but the core topic coverage is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters with descriptions, so the schema already handles parameter semantics. The description adds no extra param-specific context beyond what the schema provides, earning the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches TOTVS Protheus release notes and related topics (LIB updates, patches, behavior changes, breaking changes, deprecated functions). This is a specific verb+resource combination that distinguishes it from sibling tools like search_protheus_docs or search_community.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus the alternatives. There is no mention of exclusions, when to prefer another sibling, or when this tool is most appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deep_researchA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does disclose several behavioral traits: aggregates results, ranks by reliability, removes duplicates, and produces a detailed report. However, it omits potential side effects, dependencies, or limitations (e.g., no mention of network calls, rate limits, or that it is purely read-only).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the primary purpose and then details the output components. While the list of report elements is long, each item adds meaningful context. No wasted words, but it could be slightly more concise by trimming the exhaustive list.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description does a good job explaining the return value (a technical report with specific sections). It also covers the scope (across ALL sources) and the research process. It doesn't address how depth modifies behavior, but the schema's enum description partially covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters with descriptions, so the baseline is 3. The description adds some semantic reinforcement (e.g., 'version considerations' aligns with versions parameter), but it doesn't provide syntax, format, or usage details beyond what the schema already includes.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Orchestrate') and clearly defines the resource ('comprehensive research across ALL sources for a Protheus topic'). It distinguishes from siblings by emphasizing aggregation and report generation, unlike the targeted search tools like search_protheus_docs or search_community.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies this is the go-to tool for comprehensive research across all sources, while siblings handle specific searches. It does not explicitly state exclusions but the 'ALL sources' phrase provides clear context for when to use it over more focused alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_advpl_exampleA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description holds the burden for behavioral disclosure. It discloses quality standards ('production-ready', 'modern standards', 'no obsolete patterns') and scope (supported patterns). However, it does not disclose return format, potential failure modes, or any dependencies beyond 'researched material', leaving some ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the action, and every sentence adds value: action, supported patterns, and quality standards. No redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's scope and quality but omits what the output looks like (e.g., a code snippet, multiple examples, file structure). Since there is no output schema, this is a gap. However, given the tool's name and clear purpose, it is adequately scoped for simple generation tasks, making a 3 appropriate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers all three parameters with clear descriptions and an enum for language, so the description does not need to add much. It adds only minor context like 'SQL Server syntax', which reinforces the SQL parameter but does not meaningfully enhance understanding beyond the schema. Baseline of 3 is appropriate given 100% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's specific action ('Generate') and resource ('production-ready, modern ADVPL/TL++ code examples'), and enumerates supported patterns (ADVPL, TL++, SQL, REST, SOAP, MVC). This distinguishes it from sibling tools, which are research/search-oriented, by emphasizing code generation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'based on researched material' provides clear context that this tool should be used after research tools like deep_research or search_protheus_docs. It implicitly sets an expectation for prior information gathering, though it does not explicitly name alternatives or exclusions, which keeps it just below a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_protheus_docsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the exact data sources searched ('TDN, Central de Atendimento, Framework docs, and Release Notes') and frames the tool as authoritative, but it does not describe result format, pagination, or query limitations. Read-only behavior is implied by 'Search' rather than stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with the core action and source scope. The guidance is useful though slightly redundant ('authoritative'/'official' and repeated preference over community sources), but no sentence is wasteful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a docs-search tool with three well-documented parameters and no output schema, the description covers scope, source set, and usage policy. It could add what the response contains or how version/module filters behave, but the essential context for selecting and invoking the tool is present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds no parameter-specific meaning beyond the schema; query, module, and version are already documented with examples. 'Any Protheus-related topic' loosely frames query scope but doesn't enhance understanding of module/version filters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Search official TOTVS Protheus documentation across TDN, Central de Atendimento, Framework docs, and Release Notes' – a specific verb, resource, and source scope. The word 'official' clearly distinguishes it from sibling search_community.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this tool when you need authoritative, official TOTVS documentation for any Protheus-related topic' and 'Always prefer this over community sources for factual technical information.' It gives clear when-to-use context and contrasts with community sources, though it does not name specific sibling alternatives or negative cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_sourcesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses that the tool compares official and community sources and identifies agreements, contradictions, and version differences. However, it doesn't detail underlying source mechanisms, potential limitations, or return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences that cover purpose, behavioral outcomes, and usage context without any wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 2-parameter tool with no output schema, the description provides purpose, behavioral expectations, and usage context. It doesn't specify return format, but the 'Identifies...' phrasing gives a reasonable idea of the output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and both parameters are already described in the schema. The tool description adds no parameter-specific information, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the verb 'Compare' and the resource 'multiple information sources (official vs community) on a given topic.' It also states the key outputs (agreements, contradictions, version-specific differences), which distinguishes it from sibling search tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly provides a use case: 'Useful when documentation is conflicting or unclear.' This gives a clear context, although it doesn't mention when not to use it or alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_communityA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes what sources are searched and the type of content found, but it does not disclose expected result format, potential latency, reliability of community content, or whether network access is required. This is adequate for a read-only search but lacks some behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences and immediately states the action and key purpose. It lists sources concisely and then provides the use case. No wasted words or redundancy with the schema or title.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple search tool with 2 parameters and no output schema, the description is fairly complete: it names sources, explains the intended use, and differentiates from siblings. It does not mention the return value structure, but for a search tool this may be less critical. It also lacks caveats about community-generated content quality, but overall it covers the essential context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the parameters, so the baseline is 3. The description adds no additional meaning beyond the schema—'query' and 'includeForums' are self-explanatory in the schema. There is no extra guidance on how to craft effective queries or when to enable forums.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (search), the resource (trusted Protheus community sources), and enumerates specific sources (Terminal de Informação, BlackTDN, etc.). It also clarifies the purpose (find practical examples, workarounds, real-world experiences), which distinguishes it from sibling search_protheus_docs (docs) and search_release_notes (release notes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to find practical examples, workarounds, and real-world experiences,' providing clear context for when to invoke this tool. It does not explicitly mention when not to use it or name sibling alternatives, but the use-case phrasing conveys the intended scope compared to documentation/release searching.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• debug_protheus_errorA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that the tool combines official and community knowledge and provides diagnosis, probable causes, fixes, and prevention. This gives a clear sense of its behavior and output, though it omits details like rate limits or response format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the main purpose. Every sentence adds value: what it does, how it combines knowledge, and the supported error types. No fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and absence of annotations/output schema, the description covers the key aspects: domain, supported error families, and output categories (diagnosis, causes, fixes, prevention). It does not specify exact return format or limitations, but is adequate for an analysis tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage, clearly describing errorMessage, stackTrace, and environment. The tool description adds no extra parameter meaning, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Investigate[s] Protheus runtime errors, stack traces, AppServer logs, and common failures,' with a specific verb and resource scope. It also enumerates supported error types (Access Violations, Deadlocks, etc.), distinguishing it from sibling search tools like search_protheus_docs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when encountering Protheus runtime errors and lists supported categories, giving clear context for when it applies. However, it does not explicitly mention alternatives or exclusions, such as 'use search_protheus_docs for documentation lookups instead.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.