Server Quality Checklist

Tool Scores

• kernel_executeA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full behavioral burden. It does well: explains the exact output format (tagged blocks), the timeout behavior including the [pending] msg_id mechanism, and the blocking semantics without timeout. This is a rich disclosure of what execution does and returns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by behavioral details. Each sentence adds value: purpose, output format, timeout mechanism. Not overly verbose, but slightly longer than the minimal needed — the detail is warranted given the complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description provides substantial context: it documents the output format (tagged blocks, image blocks), timeout semantics, pending/retrieval flow, and blocking behavior. It's reasonably complete for a multi-behavior execution tool, though it could mention potential side effects (kernel state mutation) and whether auth is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the two undocumented parameters. It does explain the 'timeout' parameter's behavior in detail (exceeding returns partial output plus pending block). However, it doesn't describe the 'code' parameter beyond its obvious role, and there's no syntax/format guidance. Adequate but not comprehensive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it executes code on the running kernel and returns structured output. It identifies the verb (execute) and resource (running kernel) precisely. However, it doesn't explicitly distinguish from siblings like kernel_interrupt or kernel_get_output, though the output-format detail helps differentiate it from the other kernel_* tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (running kernel) and explains the timeout/pending behavior. However, it doesn't explicitly state when to use this tool vs alternatives, nor does it mention prerequisites like requiring kernel_start first or when to use kernel_get_output vs waiting inline. The pending-block behavior partially guides the flow but lacks explicit when/when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_stopA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful context beyond the name by stating that resources are cleaned up, which is a behavioral disclosure not otherwise provided. However, with no annotations present, the description carries the full burden for behavioral transparency. It doesn't mention that stopping is destructive/irreversible to current execution state, or what happens to pending execute calls or unsaved output, which would be valuable for an agent deciding between stop, interrupt, or restart.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tight sentence with zero waste. Every word earns its place — it states the action (stop) and the secondary effect (clean up resources). It's appropriately sized for the simplicity of the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given this is a zero-parameter tool with an existing output schema, the description conveys the essential purpose effectively. The 'clean up resources' detail adds needed behavioral context. While it could mention post-condition side effects (e.g., session state loss), the combination of a simple operation, output schema, and sibling name context makes this substantially complete for the tool's complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are 0 parameters, so the 100% schema coverage baseline applies and there is nothing for the description to clarify. With no parameters to document, the description doesn't need to add parameter semantics, and the baseline of 4 for zero-parameter tools is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a clear verb+resource combination ('Stop the running kernel') and distinguishes the stopping action from siblings like kernel_start, kernel_restart, and kernel_interrupt. It clearly states the resource being acted upon, making the purpose unambiguous even though it doesn't explicitly contrast with sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when a running kernel needs to be stopped, which is reasonably clear given the verb 'Stop' in the name. However, it doesn't explicitly state when NOT to use this (e.g., preferring kernel_interrupt for temporary pauses instead of full stop, or kernel_restart for reinitialization scenarios). The sibling differentiation is implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_discoverA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden. It discloses the return format (list of dicts with name, display_name, language, source) and the conditional behavior when scan_dir is provided. However, it doesn't disclose what happens if no kernels are found, whether errors can occur, or how the scan operates (speed, blocking behavior). The core behaviors are disclosed but with limited depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is focused and informative—roughly 5 sentences covering purpose, output format, usage linkage, and the optional parameter. It's front-loaded with the core purpose and output, then details scan_dir behavior. No wasted words, though it could be marginally tightened.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only discovery tool with one optional param and an output schema present, the description is fairly complete. It explains the return structure, links to the sibling kernel_start workflow, and clarifies scan_dir behavior. The presence of an output schema reduces the burden of explaining return values. Minor gaps remain around error/edge cases, but overall adequate for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must compensate. It does: scan_dir is explained as a path that triggers checking for a .venv with ipykernel, and the resulting 'venv:<path>' naming convention is disclosed. With one optional parameter, the description adequately covers its semantics beyond the bare schema definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (Discover) and resource (Jupyter kernel specs and project venvs). It distinguishes from siblings (kernel_start, kernel_status, etc.) by listing the return values that guide subsequent kernel_start usage. It's specific about what's returned (list of dicts with keys) and how to use the output.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear practical guidance: tells the agent to use the returned 'name' value as kernel_name for kernel_start, and explains when scan_dir matters (checks for .venv with ipykernel). This gives good context for when this discovery tool should be invoked, though it doesn't explicitly state when NOT to use it or name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_statusA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It does well by enumerating the exact dict keys returned (running, alive, project_dir, connection_file, python, pending_executions, transport, ip, shell_port, iopub_port) and explicitly noting the no-kernel case returns {'running': False}. This gives the agent a precise understanding of the output format and edge case behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured: first line states the purpose, second lists the return keys, third covers the edge case. It's front-loaded and each sentence adds value. Slightly more verbose than strictly necessary given the output schema exists, but the explicit key listing and edge-case behavior are genuinely useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having a rich output schema and zero complexity (0 params), the description goes beyond minimum requirements by enumerating all returned keys and documenting the no-kernel edge case. The behavior is fully specified for a status query with no side effects. The only minor gap is not explicitly noting this is a read-only, non-mutating operation, though that is fairly inferable from 'status'.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters and schema coverage is 100% (nothing to document). Per the rubric, 0 params earns a baseline of 4. The description appropriately spends no space on parameters since there are none, and instead uses that space to document return values, which is more valuable here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns the current kernel status, with a specific verb ('Return') and resource ('kernel status'). It distinguishes from siblings (discover, start, restart, stop, execute) since it's the only query-only status tool. However, the title is null and the verb is generic; the sibling differentiation is implied by the tool name 'status' rather than explicitly stated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a read-only status query tool but does not explicitly say when to use it vs the sibling tools like kernel_discover. For a status tool, the usage context is fairly self-evident (check state before/after operations), and the distinction from kernel_discover is somewhat ambiguous since both appear to be query-type operations. No explicit alternatives or exclusions are named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_interruptA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full behavioral disclosure burden. The description discloses the mechanism (SIGINT), the safety property (doesn't lose kernel state), and the intent (cancel long-running execution). It does not disclose return values or side effects, but the output schema exists and the zero-parameter simple behavior is fairly well covered. With no annotations, a 3 is appropriate given the moderate context provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. The first sentence states the core action, the second sentence explains when and why to use it. Every word earns its place, and the critical benefit (preserving kernel state) is front-loaded in the usage sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a conceptually simple tool (send SIGINT with no parameters) and the description covers the essential elements: what it does, when to use it, and the key behavioral guarantee. Combined with the output schema and zero-parameter simplicity, the description is reasonably complete. It could mention edge cases (e.g., what if no long-running execution exists), but for a simple interrupt tool this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, so the baseline is 4. There are no parameters to document, and the schema coverage is 100% (empty). The description correctly adds behavioral context around what the tool does even though there's nothing to parameterize.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb+resource: 'Interrupt the running kernel (send SIGINT)'. It distinguishes itself from siblings since kernel_restart, kernel_stop, and kernel_execute are all clearly different actions, and the interrupt action is unique. However, it doesn't explicitly name sibling alternatives, though the parenthetical SIGINT detail adds helpful specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: 'Use this to cancel a long-running execution without losing kernel state.' This explicitly states when to use the tool and the key benefit (preserving kernel state), which distinguishes it from kernel_stop (which would presumably destroy state) and kernel_restart. It doesn't explicitly state when NOT to use it, but the usage guidance is reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_get_outputA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses two critical behaviors: blocking/waiting semantics with timeout control, and destructive cleanup after reading ('record is cleaned up — calling again will return an error'). This is significant one-shot-consumption behavior that the agent must know. It could mention whether waiting implies any resource cost or whether partial results are returned during runs, but the provided disclosure is genuinely valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three tight paragraphs, each earning its place: purpose, behavior/timeout, and cleanup consequence. Front-loaded with the primary action in the first sentence. Zero filler or redundant restating of the tool name or schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 2-param tool with no output schema and no annotations, this description covers the critical ground: the async retrieval pattern, timeout semantics, the tag types returned, and the one-shot cleanup. It's slightly lean around edge cases (e.g., what happens on long pending runs, whether images are inlined or referenced), but for a straightforward retrieval tool the essentials are present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains msg_id implicitly as the identifier returned by kernel_execute for a pending execution, and documents the timeout parameter's role as a wait bound (or immediate return if omitted). The description adds meaningful semantics beyond the bare schema, defining both parameters' purpose and interaction.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states a specific verb+resource ('Retrieve output for a pending execution by msg_id') and clearly delineates the scope. It names the block types returned and the cleanup behavior, distinguishing it from sibling kernel_execute which presumably returns the same blocks but executes rather than retrieves.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly explains when to use this tool (after kernel_execute, when output is pending/async), the timeout behavior (waits if running, returns immediately if omitted), and the consequence of calling again after completion (error). It doesn't explicitly contrast with alternative sibling tools by name, but the async retrieval context is strongly implied through the pending execution framing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_restartA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds meaningful context: pending executions are discarded, the kernel process is replaced, the connection is preserved, and no restart is needed. It also specifies what gets cleared (variables, imports, state). This goes well beyond a bare 'Restart kernel' and discloses the destructive consequences.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences with zero waste. The first establishes purpose and scope; the second discloses key behavioral nuances (pendings discarded, connection preserved, no restart needed). Every sentence earns its place and information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with an output schema present and clear one-paragraph documentation, this is complete. It covers purpose, what gets cleared, side effects (pending executions discarded), and a key post-condition (connection preserved, no kernel_start needed). Nothing significant is missing for an agent to invoke this correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are 0 parameters, so the schema needs no explanation. The baseline for 0 params is 4, and the description appropriately doesn't clutter with parameter syntax. Since there are no parameters to document, this dimension is adequately handled.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: 'Restart the kernel, clearing all variables, imports, and state.' It uses a specific verb (restart) with a clear resource (kernel) and specifies the scope of effect (clearing variables, imports, state). Among siblings it's readily distinguishable from kernel_start, kernel_stop, kernel_interrupt, etc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clarifies that after restarting, 'no need to call kernel_start again,' which is a key usage detail distinguishing it from a full stop+start cycle. It doesn't explicitly say when-not-to-use or name alternatives, but the 'connection is preserved' note and the sibling naming convention provide adequate context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kernel_startA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses the single-kernel concurrency constraint, specifies cwd default behavior (venv project dir vs home directory), and explains the two valid kernel_name formats. It doesn't describe return value or error behavior, but the output schema exists to convey those details, so the key runtime behavioral traits are well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly structured with clear paragraph breaks: intent, kernel_name semantics, cwd semantics, and concurrency constraint. Every sentence adds value — no filler, no repetition of the schema. Front-loaded with the primary purpose and written in a scannable format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 2-parameter tool with an output schema present, this is quite complete. It covers both parameters' semantics, the single-kernel constraint, defaults, and ties the workflow to kernel_discover/kernel_stop siblings. The only gap is the absence of error handling details (e.g., what happens when the kernel name doesn't exist), but with an output schema present, that burden is partially relieved.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does. It explains kernel_name's two valid formats (registered spec names with 'python3' example, and 'venv:/path/to/project' references). It also clarifies cwd's meaning and default resolution logic. Both parameters are meaningfully elaborated beyond the bare schema fields, though it stops short of describing edge cases like empty strings or valid directory formats.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Start') and resource ('a kernel'), and specifies that the kernel is identified by name as returned by kernel_discover. It differentiates the tool's function from siblings like kernel_stop, kernel_status, and kernel_restart. The scope is well-defined with concrete examples of valid kernel_name values.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly explains when to use this tool: to start a kernel. It names the exact input source (kernel_discover) and gives a concrete when-not condition: 'Only one kernel can run at a time — call kernel_stop first if one is already running,' which properly excludes/precedes sibling kernel_stop. It also explains cwd semantics, removing ambiguity about the default behavior for different kernel types.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.