Skip to main content
Glama
sumithr

sumo-qa

by sumithr
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

sumo_qa_export_test_cases

Export structured QA test cases into deterministic Markdown, JSON, or CSV, returning rendered content by default and writing to a file only when an explicit output path is given.

Instructions

Deterministically EXPORT already-structured QA test cases into one documented machine-readable shape (issue #148). FILE/FORMAT PLUMBING ONLY — the host LLM identifies the cases; this tool never infers them and never inspects a repo. By DEFAULT it is side-effect free (it RETURNS the rendered text and writes nothing); a file is persisted ONLY when an explicit output_path is supplied.

Each case is a dict with: id (stable within this export), title, preconditions (ordered list, may be empty), steps (ordered list, may be empty), expected_result, optional linked_risk_id (a risk id in a companion risk ledger), priority (one of critical / high / medium / low), and evidence_status (one of planned / passing / failing / stale / accepted_residual — the same vocabulary as the risk ledger).

format is one of: markdown (the DEFAULT human-facing table), json (a versioned, key-sorted, deterministic document), or csv (OPTIONAL, and only valid for a flat outline — at most one precondition and one step per case). An unsupported format, or CSV for a non-flat export, returns an error envelope naming the supported formats. Tool- specific import mappings may need local adjustment.

export_title (optional) names the export as a whole — rendered in the markdown header and the JSON top-level title. (It is named export_title, not title, so it stays distinct from each case's own title.)

output_path (optional) is the EXPLICIT file-write carve-out. When omitted (the default) nothing is written. When given, the SAME rendered bytes are ALSO persisted, confined to the project export root (<cwd>/.sumo-qa/exports): a relative path resolves under that root, an absolute path or .. traversal that escapes it is refused, and an already-existing target is refused rather than silently overwritten. The write only happens AFTER successful validation+render, so a bad export never leaves a file. On a successful write written_path carries the resolved absolute path (else None).

Returns the rendered content, the chosen format, the stamped schema_version, the validated test_case_count, and written_path (the persisted location, or None on the default no-write path).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
formatNomarkdown
test_casesYes
output_pathNo
export_titleNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.42.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only give readOnlyHint=false, openWorldHint=false, idempotentHint=false, destructiveHint=false; the description adds far more — default no side effects, the explicit write carve-out, confinement to <cwd>/.sumo-qa/exports, refusal of absolute/.. traversal paths, refusal to overwrite existing targets, and validation-before-write ordering. This is exactly the behavioral context an agent needs for a conditional-write tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose then structured by parameter; each block maps to a caller decision. It is verbose and heavy with emphasis markers, and the 'import mappings may need local adjustment' line is somewhat tangential, but overall the length is justified by the complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter export tool with no output schema and no schema descriptions, the description covers purpose, format semantics, error behavior, write-path safety, and return fields (content, format, schema_version, test_case_count, written_path). Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry the full burden and it does: it documents format (markdown/json/csv with default and CSV's flat-outline restriction), the per-case dict fields (id, title, preconditions, steps, expected_result, linked_risk_id, priority, evidence_status with enumerated values), export_title's naming rationale, and output_path's resolution rules.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (EXPORT) and resource (QA test cases) plus the exact transformation: deterministic rendering of already-structured cases into a machine-readable shape. It explicitly bounds scope by stating the host LLM identifies the cases and the tool 'never infers them and never inspects a repo', so an agent knows precisely what this is and is not.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear when-to-use context: side-effect-free by default, file writes only when output_path is supplied, and CSV only valid for flat outlines. It does not name a sibling alternative or an explicit 'use X instead when...' route, but the guidance on format selection and the write carve-out is concrete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools