Server Quality Checklist

Tool Scores

• list_loaded_pdfsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool lists files and returns paths, but doesn't cover critical aspects like whether it's read-only (implied but not stated), error handling (e.g., if no PDFs are loaded), performance characteristics, or format of the returned list. For a tool with zero annotation coverage, this leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured: two sentences that state the action and the return value with zero waste. It's front-loaded with the core purpose, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, output schema exists), the description is minimally adequate. It explains what the tool does and what it returns, but lacks context about prerequisites (e.g., needing loaded PDFs) and behavioral details. With no annotations and an output schema, it meets basic needs but could be more informative for agent usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (empty schema). The description adds no parameter information, which is appropriate here. Baseline for 0 parameters is 4, as there's nothing to document, and the description correctly focuses on output instead.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List all currently loaded PDF files.' This specifies the verb ('List') and resource ('loaded PDF files'), making it immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'list_applied_redactions' or 'load_pdf', which would require mentioning it shows only loaded files (not redaction status or loading capability).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., that PDFs must be loaded first using 'load_pdf'), exclusions, or comparisons to siblings like 'list_applied_redactions'. The agent must infer usage from context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• close_pdfA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool frees resources and can raise a ToolError if the PDF is not loaded, which are useful behavioral traits. However, it lacks details on permissions, side effects, or rate limits that would be beneficial for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded with the main purpose. The structured sections (Args, Returns, Raises) are clear, but the repetition of parameter info in 'Args' slightly reduces efficiency, though it remains concise overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a simple close operation), no annotations, and the presence of an output schema, the description is fairly complete. It covers purpose, parameters, returns, and error conditions, though it could benefit from more behavioral context like resource management details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'pdf_path' parameter. The description repeats this information in the 'Args' section but does not add significant meaning beyond what the schema provides, such as format examples or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Close a loaded PDF') and resource ('free its resources'), distinguishing it from sibling tools like 'list_loaded_pdfs' or 'load_pdf'. It explicitly identifies what the tool does without being tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'a loaded PDF' and mentioning the 'Raises' condition, which indicates when not to use it (if PDF is not loaded). However, it does not explicitly name alternatives or provide detailed exclusions beyond the error case.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_pdfA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes key behaviors: loads PDF into memory, extracts text content, and keeps it loaded for future operations. However, it lacks details on memory implications, performance characteristics, or error handling beyond the basic ToolError mention.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, args, returns, raises) and uses efficient sentences. However, the parameter explanation in the description slightly duplicates the schema, and the logging context mention could be more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (loading and extracting text), no annotations, and the presence of an output schema (which handles return values), the description is reasonably complete. It covers purpose, usage context, parameters, returns, and errors, though could benefit from more behavioral details like memory usage or format constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents the single parameter 'pdf_path'. The description repeats the parameter explanation but does not add meaningful semantics beyond what the schema provides, such as file format requirements or path resolution details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Load a PDF file') and its purpose ('make it available for redaction'), distinguishing it from sibling tools like 'list_loaded_pdfs' (which only lists) or 'redact_text' (which modifies). It explicitly mentions the resource (PDF file) and the outcome (extracts text content for review).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: to load a PDF for subsequent redaction operations. It implies usage by mentioning that the PDF remains loaded for later steps, but does not explicitly state when not to use it or name alternatives like 'list_loaded_pdfs' for checking already loaded files.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• redact_areaA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by explaining key behavioral traits: it's a preparatory step (redactions not applied yet), requires a loaded PDF, has specific error conditions (PDF not loaded, page doesn't exist), and returns a confirmation message. It doesn't cover rate limits or auth needs but provides substantial operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, behavior, parameters, returns, errors) and appropriately sized. The first sentence immediately states the core purpose. Some redundancy exists between parameter descriptions and schema, but overall it's efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (7 parameters, mutation operation) and no annotations, the description provides good coverage: explains purpose, behavior, parameters, returns, and error conditions. With an output schema present, it doesn't need to detail return values. It could mention sibling tools more explicitly but is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds minimal value beyond the schema by briefly explaining coordinate parameters and the default fill_color, but doesn't provide additional semantic context like coordinate system details or practical usage examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('redact a specific rectangular area'), resource ('on a PDF page'), and distinguishes from siblings by noting redactions are not yet applied, differentiating from save_redacted_pdf. It provides a precise verb+resource combination.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('adds a redaction annotation') and when not to use it ('redactions are not yet applied - use save_redacted_pdf to apply and save'). It provides clear context but does not mention alternatives like redact_text or prerequisites beyond PDF loading.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• redact_textA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: the redactions are annotations not yet applied, it only processes unredacted texts, requires a loaded PDF, and mentions error conditions (ToolError if PDF not loaded or redaction fails). It doesn't mention rate limits or authentication needs, but covers the essential operational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It front-loads the core functionality in the first sentence, then provides operational details, prerequisites, and relationships to other tools. Every sentence earns its place with no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no annotations, but with output schema), the description is complete enough. It explains what the tool does, when to use it, behavioral constraints, parameter purposes, and error conditions. The existence of an output schema means the description doesn't need to detail return values, and it adequately covers the operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds minimal value beyond the schema: it clarifies that texts_to_redact searches for 'all instances' and that fill_color is for 'the redaction box,' but doesn't provide additional semantic context like text matching behavior (case sensitivity, partial matches) or color interpretation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('redact specific texts', 'searches for all instances', 'adds redaction annotations') and resources ('in a loaded PDF'). It distinguishes itself from siblings by focusing on text-based redaction versus area-based (redact_area) and specifying it's not the saving operation (save_redacted_pdf).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: 'Only texts that haven't been previously redacted will be processed.' It also specifies when not to use it (for saving) by directing to 'use save_redacted_pdf to apply and save.' It clearly distinguishes from sibling tools like redact_area (text vs area) and save_redacted_pdf (annotation vs application).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• save_redacted_pdfA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the mutation behavior ('apply all redactions and save'), default file naming convention, error conditions ('PDF not loaded or save fails'), and return value. However, it doesn't mention side effects like whether the original PDF is modified or if redactions are cleared after saving.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and front-loaded: the first sentence states the core purpose, followed by elaboration, parameter details, return value, and error handling. Every sentence adds value with zero redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (applying redactions and saving), no annotations, but a complete input schema (100% coverage) and an output schema (implied by 'Returns'), the description is fully adequate. It covers purpose, usage, parameters, returns, and errors, leaving no gaps for the agent to understand and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds meaningful context beyond the schema: it clarifies that 'pdf_path' refers to an 'already loaded' PDF (implying a prerequisite), explains the default naming convention for 'output_path' in more detail, and mentions 'ctx' as a logging context (though not in the schema). This extra semantic value justifies a score above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('apply all redactions and save') and resource ('the redacted PDF'), distinguishing it from siblings like 'redact_area' or 'redact_text' which only create redactions, and 'list_applied_redactions' which only lists them. The verb+resource combination is precise and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('apply all pending redaction annotations') and mentions prerequisites ('PDF must be already loaded'), but does not explicitly state when not to use it or name specific alternatives among the siblings. The context is helpful but lacks explicit exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_applied_redactionsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses behavioral traits: it lists applied redactions, helps avoid duplicates, tracks processed items, and raises ToolError for unloaded PDFs. However, it doesn't mention performance aspects like rate limits or detailed error handling beyond ToolError.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loaded with the core purpose, followed by benefits, args, returns, and raises sections. Every sentence earns its place: no fluff, efficiently structured with clear sections for different aspects of the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (1 parameter, no nested objects), no annotations, but with an output schema (so return values are documented elsewhere), the description is complete. It covers purpose, usage, parameters, returns, and error conditions adequately for a listing tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the optional nature of pdf_path ('If not provided, lists all PDFs') and the context parameter ('MCP context for logging'), which clarifies usage beyond the schema's technical details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and resource 'redactions that have been applied to loaded PDF(s)', specifying it shows which texts have been marked for redaction. It distinguishes from siblings like list_loaded_pdfs (which lists PDFs, not redactions) and redact_area/redact_text (which apply redactions, not list them).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'helping to avoid duplicate redactions and track what has already been processed.' It implies when not to use (e.g., for applying redactions, use redact_area/redact_text instead) and provides context about loaded PDFs, though it doesn't name alternatives directly beyond the sibling context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.