Server Quality Checklist

Tool Scores

• delete_fileB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It only states the action without noting whether deletion is permanent, required permissions, or side effects. This is insufficient for a destructive operation, leaving significant behavioral uncertainty.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, highly concise and front-loaded. It contains no unnecessary words or repetition, earning a perfect score for efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only 2 parameters, but given the absence of annotations, output schema, and parameter explanations, the description provides only the bare minimum. It does not mention return values, error behavior, or how this tool fits with siblings like compile or get_log. It is minimally viable but lacks contextual depth.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 2 required parameters (path, projectId) with no descriptions (0% coverage). The tool description does not mention or explain these parameters, providing no additional meaning beyond the parameter names. The description fails to compensate for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Delete a file from the project' uses a specific verb and resource, making the tool's function unambiguous. It clearly distinguishes from sibling tools like read_file, write_file, and list_files by indicating the delete operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives, nor does it mention prerequisites or conditions. Usage is implied by the tool name and simple statement, but no exclusions or alternative recommendations are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It adds a meaningful nuance: the listing reflects edits made by humans in the web UI, not just server-written files. While it doesn't explicitly state read-only behavior, the verb 'List' plus the non-mutating context implies a safe operation. The added context about human edits is valuable beyond what a simple tool description provides.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no fluff. It front-loads the core purpose in the first sentence and adds a valuable behavioral note in the second. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description gives the essential behavior and even an extra data-source nuance, but with no output schema and no mention of return format (beyond 'size'), error conditions, or whether directories are included, some gaps remain. For a simple one-parameter tool, it is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameters. It does not mention projectId at all. The only clue is the phrase 'in the project', which vaguely implies projectId identifies the project, but no actual guidance is given. This fails to add meaning beyond the bare schema field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists every file in the project with its size, which is a specific verb+resource action. This distinguishes it from sibling tools like read_file or write_file, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for obtaining a full file listing with sizes, but does not explicitly state when to use this tool versus alternatives like read_file for individual files or delete_file for removal. It gives no explicit exclusions or alternative suggestions, so guidance is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_logA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It notes the log is raw and long, hinting at potentially large output. However, it omits error conditions, behavior when no compile has run, and what maxBytes controls, leaving gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action, and no filler. 'the raw log is long' efficiently justifies the usage constraint and earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 2 parameters and no output schema, but with 0% schema coverage and no annotations, the description needs to explain parameters and edge-case behavior. It misses what happens before the first compile, maxBytes semantics, and possible error responses, so the description is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description names no parameters. The optional maxBytes parameter is completely undocumented, and projectId is only implied by context. The description adds zero value beyond the schema, which itself is terse.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'Fetch the raw LaTeX log from the last compile' with a specific verb and resource. Distinguishes from sibling tools like compile (which provides diagnostics) and read_file (which reads files generally).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Use only when compile's structured diagnostics are not enough' explicitly prescribes when to use this tool and points to the alternative (compile). This provides clear decision guidance, similar to the TDQS example that names an alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_projectA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It usefully notes that the project is immediately visible and editable in the Overleaf web UI by a human, but it omits important details such as permission requirements, name-conflict behavior, or potential side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two front-loaded sentences. The first states the core action and return value, and the second adds useful contextual information. There is no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter creation tool, the description adequately covers the action, the immediate side effect (visibility/editable in the web UI), and the return value (id). It does not specify the exact return format or conflict behavior, but given the tool's simplicity and full schema coverage, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers the single 'name' parameter at 100% with a clear description. The tool description adds no parameter-specific meaning beyond what the schema already provides, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Create') with a clear resource ('a new Overleaf project') and states the return value ('return its id'). This clearly distinguishes it from the sibling file-manipulation and compilation tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose is inherently distinct from sibling tools, implying this should be used when a new project is needed. However, it does not explicitly state when to use it versus alternatives, nor does it mention any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_fileA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It correctly implies a read-only operation but does not describe return format, error behavior (e.g., file not found), or any side effects. The note about human-touched files adds context but not tool mechanics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the core action. Every word earns its place; no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter read tool, the description provides purpose and a clear usage hint. However, without an output schema or mention of return value, the agent may need to infer what the tool returns, which is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50% (path has an example, projectId does not). The description does not explain projectId or add meaning beyond the schema's path example. It fails to compensate for the missing parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Read') and resource ('one file from the project'), clearly distinguishing it from sibling tools like list_files and write_file. Purpose is immediately obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Call this before rewriting a file a human may have touched, so your edit is based on current content.' This tells the agent when to use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fileA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits on its own. It reveals operational transform merging, live document updates, cursor preservation, and whole-file replacement—all valuable context beyond mere 'write.' It does not mention permissions or failure modes, but the disclosed details are significant.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences plus a prominent NOTE, with a clear front-loaded purpose. It wastes no words and structures the key safety warning at the end, making it easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (operational transform, live updates, overwrite risk), the description covers the critical behaviors and safety guidance. It does not describe return values, but no output schema exists, and this is not essential for a write operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 33% (only 'path' has a description). The description adds no explicit meaning for 'content' or 'projectId,' though the note about replacing the whole file implies 'content' is the full file contents. This is insufficient compensation for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with 'Create or overwrite a file,' which clearly states the verb and resource. This distinguishes it from sibling tools like read_file and delete_file by focusing on the write operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly advises to 'read_file first if the content may have changed,' providing a direct alternative and a safety caveat. It implies this is the tool for creating or modifying file contents, but does not fully enumerate when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_pdfA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that it compiles, saves to a local file, and returns the path, which are key behaviors. However, it omits potential side effects like overwriting files or permission requirements, so it is not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each purposeful: the first states the function, the second gives usage guidance. No redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 2 parameters and no output schema, the description covers the core purpose, usage guidance, and return value. It lacks parameter details (already penalized) and error handling, but provides a sufficient high-level understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only covers outputPath with a default note, leaving projectId undescribed. The description adds no parameter-specific meaning, failing to explain what projectId identifies or how outputPath behaves beyond the schema, despite the low 50% coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'compile and save' with a clear resource (PDF) and specifies the outcome (local file path). It also distinguishes from sibling 'compile' by indicating when to use each, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states 'Use when you need the artifact itself; for fixing errors use compile instead.' This provides clear context and names an alternative tool, fulfilling the when-to-use guidance thoroughly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compileA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and discloses return shape (verdict, errors with file/line and suggestions), an important limitation (PDF is not returned to avoid context flood), and the BLOCKED verdict's irreversibility. This goes well beyond basic semantics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: purpose/result, critical exclusion (PDF), and verdict handling. No redundancy, front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no annotations or output schema, the description covers return format, failure modes (BLOCKED), and an alternative tool (get_pdf). This gives the agent sufficient context to use and interpret the tool's response.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50%; timeoutSec and rootResourcePath are already described in the schema. The description adds no parameter-specific meaning, but the remaining parameters (compiler enum, projectId) are self-explanatory in context. It neither clarifies nor harms.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Compile') and resource ('the project'), and clearly states the return type ('structured diagnostics'). It distinguishes from sibling tools like get_pdf (which returns the PDF bytes) and get_log.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when not to use: 'Does NOT return the PDF itself — use get_pdf if you need the bytes.' Also instructs to 'Check verdict first: if it says BLOCKED, the server environment is at fault and retrying will not help,' which clarifies a key usage nuance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.