Server Quality Checklist

Tool Scores

• quarto_create_projectA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well: it discloses that the project is temporary, that the returned projectId is opaque and process-lifetime-bound, and that initial files are written. This is meaningful beyond the schema's static property descriptions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences with no filler: the core action, the lifespan caveat, and the return value are front-loaded. Every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a creation tool with five parameters fully described in the schema, the description covers the essential operational context: temporary lifecycle, initial file writes, and a projectId handle. It could optionally specify the response shape or cleanup semantics, but the described contract is sufficient for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents every parameter. The description adds context around 'initial files' and return but does not need to restate parameter-level semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Create'), a concrete resource ('Quarto project'), and clarifies behavior ('temporary', 'write initial files', returns an opaque projectId). This clearly distinguishes it from the sibling tools, which fetch defaults, render, inspect, or set defaults.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description communicates what the tool does and that it is for creating a project, but it does not explicitly state when to choose it over siblings or when not to use it. The intended use is implied rather than directly contrasted with alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• quarto_defaults_setA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It does well by stating that the call 'replaces the whole file' and that sending empty metadata clears the defaults, which communicates the destructive/overwriting nature. It could add more about scope or side effects, but the critical behaviors are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. It front-loads the purpose and then gives the two most important behavioral details: whole-file replacement and clearing via empty metadata.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter setter with no output schema, the description covers purpose, replacement semantics, and clearing behavior. It is largely complete, though it does not explicitly state that existing projects are unaffected or describe any return value, but those are minor for this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the metadata parameter and the replacement/clearing behavior. The description repeats the examples but does not add new semantic detail beyond what the schema provides, so a baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb and resource: 'Store Quarto metadata... for every new project,' and adds the key behavioral detail that the call 'replaces the whole file.' This distinguishes it from the sibling get tool and other project/render tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for every new project' gives some context for when this tool applies, but the description does not explicitly state when to use this tool versus quarto_defaults_get or when not to use it. There are no exclusions or alternatives named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• quarto_defaults_getA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full disclosure burden. It does this well by specifying what is returned (metadata plus file path) and disclosing the absent-file edge case ('returns empty metadata'). The only unstated aspect is potential failure modes beyond absence, but for a read tool this is strong coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences with no filler. The first sentence front-loads the core action and return values, and the second efficiently disposes of the no-file edge case. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter read tool with no output schema and no annotations, the description covers the essentials: what is retured, the file involved, and behavior when absent. It could additionally point to quartz_defaults_set as the mutation counterpart, but that omission is minor given how self-contained the tool is.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the parameter-semantics burden is trivial. Per the baseline for 0-param tools this scores 4: there is nothing to document, and the description does not need to add parameter-level meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Return'), a specific resource (the Quarto metadata defaults stored as _metadata.yml), and explicitly states the two return elements: the metadata and the path of the holding file.The resource and read-only nature clearly distinguish it from siblings like quartz_render, quartz_create_project, and the write-counterpart quartz_defaults_set.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The read behavior is implied by the name and description, so an agent can infer this tool is for reading defaults, but there is no explicit guidance about when to choose it over quartz_defaults_set or other siblings, and no exclusions or alternative routing are stated. Usage context is inferable but never made explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• quarto_inspectA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It adds a useful behavioral guarantee: 'The output is passed through without field changes,' and clarifies the project-versus-file scope distinction. It does not explicitly say 'read-only' or 'no side effects,' but 'Return ... inspect JSON' plus the passthrough guarantee strongly imply a non-mutating inspection operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short, front-loaded sentences deliver the primary action and scope first, then a precise output behavior guarantee. Every sentence earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, read-style tool with fully documented parameters, the description covers what the tool returns, its scope, and pass-through behavior. Without annotations or an output schema, it could go further by explicitly stating that no project modifications occur, but the inspect semantics and passthrough statement make the behavior sufficiently clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% parameter documentation, including descriptions for projectId and input. The description adds no further parameter-level detail beyond the project-versus-file distinction, so the schema carries the weight and the description meets the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Return') and resource (

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the use case clear by indicating the tool returns Quarto's inspect JSON for either a project or a specific input file, which distinguishes it from siblings like quarto_create_project, quarto_render, and quarto_defaults_get/set. It does not explicitly name alternatives or state when not to use it, but the context is unambiguous enough for an agent to select it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• quarto_renderA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It usefully discloses that code execution is off unless 'execute' is true and that Quarto failures are reported as success:false while preserving stdout and stderr. These are material behavioral details that go beyond the schema, though it does not cover every possible side effect of rendering.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The core action is front-loaded, and the two most important caveats (execution off by default, failure behavior) are stated directly and efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a rendering tool with five parameters, the description covers the main choices an agent needs to make: what to render, whether to execute code, and how failures surface. Since there is no output schema, more return-value detail could be helpful, but the failure semantics are disclosed and the invocation context is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents each parameter. The description adds little beyond echoing the 'input' file vs project distinction already present in the schema, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Render') and a clear resource ('the project, or one input file'), making the tool's action unambiguous. It is clearly distinguishable from the sibling tools, which cover creation, inspection, and defaults handling rather than rendering.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use the tool: render either the whole project or a single input file. It does not explicitly name alternatives or state when not to use it, but the sibling tools are semantically distinct enough that an agent can infer the appropriate selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.