run_spec
Execute YAML test specifications against MCP servers to validate functionality and return detailed results for reliability testing.
Instructions
Run a YAML test spec against an MCP server and return results. Provide either specText (inline YAML) or specPath (path to a file). At least one is required.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| specText | No | Inline YAML spec content to run | |
| specPath | No | Path to a YAML spec file to run | |
| headers | No | HTTP headers (for future use — headers typically come from the spec itself) | |
| timeoutMs | No | Timeout in milliseconds (default: 30000) |
Implementation Reference
- src/tools/runSpec.ts:40-69 (handler)The `runSpec` function executes the logic for the "run_spec" tool. It takes `RunSpecInput` (spec text or path), runs the CLI command, parses the JSON output, and returns the result with a formatted summary.
export async function runSpec(input: RunSpecInput): Promise<RunSpecOutput> { if (!input.specText && !input.specPath) { throw new Error( "Either specText or specPath must be provided to run a spec.", ); } let report: RunReport; if (input.specText) { report = await withTempFile(input.specText, async (tmpPath) => { const args = ["run", tmpPath, "--json"]; if (input.timeoutMs !== undefined) { args.push("--timeout", String(input.timeoutMs)); } const result = await runCli(args, { timeoutMs: input.timeoutMs }); return JSON.parse(result.stdout) as RunReport; }); } else { const args = ["run", input.specPath!, "--json"]; if (input.timeoutMs !== undefined) { args.push("--timeout", String(input.timeoutMs)); } const result = await runCli(args, { timeoutMs: input.timeoutMs }); report = JSON.parse(result.stdout) as RunReport; } const text = formatReport(report); return { text, structured: report }; } - src/server.ts:119-146 (schema)The MCP tool definition for "run_spec" including its input schema and description.
{ name: "run_spec", description: "Run a YAML test spec against an MCP server and return results. Provide either specText (inline YAML) or specPath (path to a file). At least one is required.", inputSchema: { type: "object" as const, properties: { specText: { type: "string", description: "Inline YAML spec content to run", }, specPath: { type: "string", description: "Path to a YAML spec file to run", }, headers: { type: "object", additionalProperties: { type: "string" }, description: "HTTP headers (for future use — headers typically come from the spec itself)", }, timeoutMs: { type: "number", description: "Timeout in milliseconds (default: 30000)", }, }, }, }, - src/server.ts:237-243 (registration)The request handler in `server.ts` that dispatches the "run_spec" tool request to the `runSpec` implementation.
case "run_spec": { const output = await runSpec( args as unknown as Parameters<typeof runSpec>[0], ); return { content: [{ type: "text" as const, text: output.text }], };