Server Quality Checklist

Tool Scores

• download_assessment_pdfA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses the attempt via a specific endpoint and fallback on error, but lacks details on permissions, rate limits, or whether it's read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with action, no wasted words. Concisely conveys purpose and context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Description covers the core action and fallback but does not explain return values or output format. For a simple download tool, this is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds no additional parameter meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: downloading a PDF assessment via a specific REST endpoint. It differentiates from the broken UI export and from siblings like generate_remediation_pdf by focusing on the assessment PDF.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives context for when to use (when UI export is broken) and notes graceful fallback, but does not explicitly state when not to use or compare with alternatives like generate_remediation_pdf.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_graphqlA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It mentions 'arbitrary query/mutation' implying potential mutability but lacks details on authentication, rate limits, or side effects. The term 'escape hatch' is vague regarding behavioral boundaries.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences deliver purpose and usage context without any redundant or extraneous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (2 params, no enums, no output schema). The description covers core purpose but does not explain what the response looks like (e.g., GraphQL result structure), which may be needed since there is no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with both parameters described. The description does not add additional semantic value beyond the schema, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Run', the resource 'GraphQL query/mutation against the NowSecure Platform API', and its role as an 'escape hatch'. It effectively distinguishes from sibling tools (e.g., download_assessment_pdf) which are specific output generators.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Escape hatch for schema introspection and custom queries' provides clear context for when to use this tool. However, it does not explicitly indicate when not to use it or provide alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_applicationsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the endpoint and the purpose, but does not detail pagination, ordering behavior, rate limits, or the complete return structure. The parameters hint at pagination but are not explicitly described in behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the purpose, and contains no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list tool with simple parameters, the description is fairly complete. It explains the output (refs and latest assessment) and the REST endpoint. However, it lacks details on the response format, which could be helpful without an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents both parameters. The description adds the context that the list contains 'app refs and latest assessment', which provides some context but does not add new meaning to the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'List', the resource 'applications in your NowSecure portfolio', includes the REST endpoint, and specifies the output 'app refs and the latest assessment per app'. It distinguishes from sibling tools like download_assessment_pdf and run_graphql by focusing on listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'Use to discover app refs and the latest assessment per app', giving direct usage context. It does not mention when not to use or provide alternatives, but the sibling tools imply different use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_remediation_pdfA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that the PDF is rendered locally, includes only open findings with default severities, guarantees to work, and auto-names files. It lacks details on authentication or return value, but is fairly transparent about core behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences, each adding essential information: purpose with reliability assurance, inclusion criteria, and naming convention. No redundancy or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters and no output schema, the description lacks mention of the return value (e.g., file path or success status) and preconditions like requiring an assessment with remediation findings. It is functional but not fully complete for an agent to infer all side effects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, baseline 3. Description adds value by explaining the auto-naming pattern (outputDir + appName), that outputPath overrides it, and the default value of impactTypes as an array of severities. This goes beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates a remediation PDF from NowSecure findings, specifying it's a clean local PDF that avoids the broken NowSecure service. It distinguishes from general assessment PDFs by focusing on open findings requiring remediation with default severities listed.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context for when to use this tool: when the UI/REST PDF export fails, and for generating a remediation-focused PDF with specific severity levels. However, it does not explicitly mention when not to use it or name alternative tools like download_assessment_pdf for full assessments.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_remediation_findingsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that only open findings (status detected/fail/open) are returned, filtered by severity with a default list, and that passed/dismissed findings are excluded. It also mentions bypassing the broken PDF renderer.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three efficient sentences with no wasted words. The first sentence front-loads the core purpose, followed by context and filtering details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters and no output schema, the description is quite complete, explaining return content and filtering. It could be slightly improved by describing the output JSON structure, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with all parameters described. The description adds context by explaining the purpose of 'appRef' and 'assessmentRef', and that 'impactTypes' are severities with defaults, augmenting the schema's descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool pulls findings that need remediation for an assessment as structured JSON. It distinguishes from sibling tools like 'download_assessment_pdf' and 'generate_remediation_pdf' by mentioning it bypasses the broken UI PDF renderer and queries GraphQL directly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use it: to get structured JSON of open findings requiring remediation, excluding passed/dismissed. It implies alternatives for PDF output but does not explicitly state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.