Server Quality Checklist

Tool Scores

• ingest_new_documentsB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits; it only mentions forcing re-ingestion without covering side effects, permissions, or handling of the force_reindex flag. This leaves significant ambiguity about the tool's impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, using two short sentences that quickly convey the purpose and context without unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no annotations and the description fails to cover parameters or side effects, leaving the 2-parameter tool under-described even though an output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage and the description does not explain the 'path' or 'force_reindex' parameters, offering no semantics beyond the raw schema field names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: forcing re-ingestion of new or modified PDFs, which distinguishes it from the read-oriented sibling tools (search, list, statistics). The verb and resource are specific, and it notes the automatic vs manual context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides context that ingestion is normally automatic via a watcher but can be manually invoked, indicating when to use it. However, it does not explicitly name alternative tools or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cross_topic_searchB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It does not mention whether the operation is read-only, what the output structure looks like, or any side effects. It merely states the basic search action, leaving the agent to infer safety and behavior from the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, with the key purpose in the first sentence and an added usage hint in the second. It avoids redundancy but is somewhat minimal, though this is not a significant flaw.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 3 parameters, no schema descriptions, and no annotations, the description is insufficient for full contextual understanding. It lacks parameter semantics, explicit alternative guidance, and behavioral details. The presence of an output schema helps but does not compensate for these gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description maps 'tema' to query and 'múltiplas disciplinas' to disciplines, but it does not mention the top_k_per_discipline parameter. With 0% schema description coverage, the description only partially compensates for missing parameter documentation, leaving a key parameter vague.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a cross-topic search across multiple disciplines with a specific verb ('busca transversal') and resource (theme in multiple disciplines). It distinguishes itself from sibling tools like search_documents by focusing on cross-discipline concept connection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that the tool is useful for connecting concepts across different course modules, providing clear context for when to use it. However, it does not explicitly contrast it with sibling tools or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_documentsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that the tool returns relevant excerpts with document and page references, which is helpful. However, it does not mention any limitations, error conditions, or the read-only nature, leaving some behavioral aspects implicit.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences plus a compact argument list. It front-loads the main action ('Busca semântica') and avoids redundant wording. Every sentence serves a clear purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core purpose, parameter meanings, and return summary (excerpts with document/page). An output schema exists, so full return structure is handled there. It lacks guidance on when to prefer sibling tools, but for a simple search tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by providing a clear Args block explaining each parameter: query (question/topic), top_k (number of results, default 5), and discipline (optional filter). This adds semantic meaning beyond the bare schema types and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Busca' (searches) and identifies the resource 'acervo de PDFs' (PDF collection), also noting the semantic nature of the search. It is clear what the tool does, but it does not explicitly differentiate from the sibling 'cross_topic_search', so a 4 is given instead of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: when a semantic search over the PDF collection is needed. However, it provides no explicit when-not-to-use guidance or references to alternatives like 'cross_topic_search'. The optional 'discipline' filter suggests context, but no exclusions or prerequisites are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_index_statsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It indicates a read operation via 'Retorna' and lists specific statistics returned, but does not address potential side effects, authentication, or rate limits. The description is adequate but minimal for a read tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, front-loaded with the main action ('Retorna'), and contains no unnecessary words. Every part adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with no parameters and an output schema. The description clearly states the high-level return values (total documents, chunks, last update), which is sufficient for the tool's complexity. The output schema presumably provides structural details, so the description doesn't need to elaborate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description needs no parameter-level information. Per the rubric, a baseline of 4 applies when there are no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Retorna' (returns) and identifies the resource as 'estatísticas do índice' (index statistics), clearly distinguishing it from sibling tools that search, list, or ingest documents. The scope is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives. The description simply states what it does without mentioning any comparative context or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_documentsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the core behavior (listing indexed PDFs) and the optional filter, but does not mention any edge cases, performance implications, or that it is a read-only operation. This is minimal but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, compact sentence that is front-loaded with the main action. It conveys the essential information without any redundancy or unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple tool with one optional parameter and an output schema. The description adequately explains what it does and the filtering option. While it lacks explicit differentiation from sibling tools, that gap is covered under usage guidelines. For its complexity, it is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has only one parameter 'discipline' with no description. The description adds meaning by stating it is an optional filter, thereby explaining the parameter's purpose. It does not specify allowed values or format, but the term 'disciplina' is self-explanatory in context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all indexed PDFs with an optional filter by discipline. The verb 'Lista' and resource 'PDFs indexados' are specific, and the scope 'todos' distinguishes it from sibling search tools like search_documents or cross_topic_search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: it is for listing all PDFs, optionally narrowed by discipline. However, it does not explicitly mention when to choose this over sibling tools like search_documents or get_index_stats, nor does it state any exclusions. The usage context is implied rather than directly specified.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.