Skip to main content
Glama
qa-toolkit-mcp

qa-toolkit-mcp

by darizae

🧭 Evidence-Driven QA

From API surfaces to test stories to traceable evidence.

Discover · Question · Specify · Index · Audit

PyPI Python License: MIT MCP Registry Agent Skills Open Source

An open-source Python CLI, local MCP server and portable Agent Skill for systematic QA planning.

Quick start · The methodology · Agent setup · MCP setup · Documentation


✦ The idea

Great integration tests begin before writing test code. First map the actual API surface. For every operation ask: who may do this, who must be denied, what conditions and relationships must hold, what changes, and what observable result would prove it? Convert the answers into test stories, assign durable case IDs, and then inspect the existing test suite.

    OPENAPI / SOURCE CONTRACT
              │
              ▼
   ① DISCOVER TESTABLE SURFACE
      endpoints · actors · states
              │
              ▼
   ② ASK & ANSWER QUESTIONS
      allowed? forbidden? conditions?
      relationships? effects? proof?
              │
              ▼
   ③ DESIGN TEST STORIES
      positive · negative · edge cases
              │
              ▼
   ④ CREATE INDEXED CASES
      stable IDs · expected evidence
              │
              ▼
   ⑤ AUDIT EXISTING TESTS
      reference matching (not a PASS)
              │
              ▼
   ⑥ IMPLEMENT → EXECUTE → REPORT
         planned next stages

The design is agent-assisted, deterministic at the tool boundary. AI agents reason about semantics and ask for missing product decisions. The Python core extracts the surface, validates reviewed stories, generates consistent case IDs and searches existing test sources. It never guesses an authorization policy from OpenAPI or treats a matching test ID as proof of a passing test.

Related MCP server: swagger-mcp

✦ Quick start

Install the package (Python 3.11+):

pip install evidence-driven-qa

Or run without a permanent installation using uvx:

uvx --from evidence-driven-qa qa-toolkit --help

1 · Discover the HTTP API surface from a saved local OpenAPI JSON file:

qa-toolkit surface --openapi openapi.json --json > qa-surface.json

This produces an editable per-endpoint questionnaire: declared routes, input parameters, response codes, available testing topics, and questions about intended behavior.

2 · Answer the contract questions and write test stories. Give qa-surface.json and the application's requirements to your agent. Ask it to complete the contract and stories fields using evidence from your project's specification, and to flag ambiguous requirements. Review its answers before compiling them.

3 · Create indexed test cases:

qa-toolkit cases --answers qa-surface.json > INDEXED_TEST_PLAN.md

Or use --json to produce a machine-readable registry. Generated IDs are deterministic for each method + path + story key and every new case starts as specified_unverified.

4 · Audit the tests that already exist:

qa-toolkit audit --plan INDEXED_TEST_PLAN.md --tests tests/ --json

The audit finds potential matches in pytest/Vitest/Playwright test source files. A reference is marked referenced_unverified, not covered or passed. Assertion-level review and executed test evidence are separate stages.

Try the independent synthetic example: example API contract, OpenAPI file, reviewed answers. These are fictitious fixtures, not production data.

qa-toolkit surface --openapi examples/synthetic-api/openapi.json
qa-toolkit cases --answers examples/synthetic-api/answers.json

✦ How it works

The workflow is endpoint-by-endpoint, but not merely endpoint-by-endpoint testing. Resource relationships, states, and actors matter as much as HTTP routes.

Step

Questions the agent answers

Output

Surface discovery

What endpoints, verbs, parameters, responses and security declarations exist?

OpenAPI inventory

Intended behavior

What user/business capability should this operation provide?

Requirement-backed contract

Allowed vs forbidden

Who is permitted? Who must be denied, including anonymous and foreign owners?

Actor/access matrix

Conditions & relationships

Which resource IDs, ownership, tenant, parent/child relations and lifecycle states must hold?

Test-data dimensions

Side effects & observability

What changes or stays unchanged? How do we prove that from HTTP and persistent state?

Concrete assertion targets

Stories & case IDs

Which positive, forbidden, invalid, missing and transition scenarios expose a meaningful bug?

Stable indexed cases

Audit & evidence

Which existing tests actually assert these outcomes and which have run?

References now; verified execution planned

More than security testing

Test stories can target authentication and authorization, input validation, business rules, resource isolation, information disclosure, pagination and filtering, async queues and timeouts, idempotency and retries, concurrency, data integrity, error contracts, rate limiting, exports, compatibility, accessibility and failure recovery. Select applicable dimensions—not a mechanically generated Cartesian product of every possible input.

For example, a negative authorization story may require checking both the rejected status and that the response reveals nothing about a foreign record. For an asynchronous job, the required proof may include a real queued state followed by an eventual completion in a disposable environment. See the complete methodology.

✦ Agent setup

The skill supplies a reusable workflow for discovering contract gaps and writing meaningful, indexed test stories. It can work with any agent that supports the Agent Skills standard.

Install to supported coding agents:

npx skills add darizae/qa-toolkit --skill evidence-driven-qa

Or install manually:

# Claude Code — user-level
mkdir -p ~/.claude/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md ~/.claude/skills/evidence-driven-qa/

# Codex / cross-agent project-level
mkdir -p .agents/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md .agents/skills/evidence-driven-qa/

Suggested first agent instruction:

Use the evidence-driven-qa skill. Discover the testable API surface, inspect the requirements for each endpoint, answer who is allowed or denied and under which conditions, then design positive/negative/edge test stories. Produce stable case IDs and audit existing tests. Call out any unresolved business rules; do not assume implementation behavior is the intended contract.

Model and coding-agent ecosystem

The toolkit does not depend on a particular language model. Via compatible coding agents or MCP-capable hosts, it is suitable for workflows using these common model families:

Model family

Example agent/host

OpenAI GPT models (GPT-6, GPT-5 series, GPT-4o)

Codex and other MCP-aware clients

Anthropic Claude (Opus, Sonnet, Haiku)

Claude Code, Claude Desktop

Google Gemini (Pro, Flash)

Gemini CLI and compatible agent hosts

Meta Llama, Qwen, Mistral, DeepSeek

Local or hosted models through MCP-capable agent runtimes

xAI Grok

MCP-capable agent integrations

MCP integration is provided by the host or agent runtime, not by model weights themselves. The generic skill can also be read as instructions by agents without MCP support.

✦ MCP server setup

The Python distribution also provides qa-toolkit-mcp, a local, read-only stdio server using the same core as the CLI. No hosted account, API key, or commercial license is required.

Claude Code

claude mcp add --transport stdio qa-toolkit -- \
  uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/project

Codex CLI

codex mcp add qa-toolkit -- \
  uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/project

Generic MCP stdio client configuration (replace the project path):

{
  "mcpServers": {
    "qa-toolkit": {
      "command": "uvx",
      "args": [
        "--from", "evidence-driven-qa",
        "qa-toolkit-mcp",
        "--project", "/absolute/path/to/your/project"
      ]
    }
  }
}

The MCP server exposes four read-only tools:

Tool

Purpose

qa_capabilities

List operations and current evidence limits

qa_discover_api_surface

Load a local OpenAPI JSON file and return the questionnaire

qa_compile_indexed_cases

Compile reviewed answers into indexed case records

qa_audit_case_references

Match case IDs against test-source references

All paths are restricted to the selected project root. The MCP tools do not launch applications, run shell commands, modify files, access credentials, or connect to external services. The local stdio package is distinct from a hosted HTTPS ChatGPT app.

✦ Open-source by design

MIT-licensed. Free to use, study, modify, fork, redistribute and build upon—including commercial use under the terms of the license. The MCP server is part of the same open-source project; no subscription or paid account is needed.

Contributions welcome: bug reports, adapters, security improvements, documentation and reproducible examples. Open an issue before substantial API or schema changes.

This is an early release of the discovery and planning workflow. Test generation into project-native source, controlled test execution, browser evidence ingestion and HTML reports are on the roadmap. No current command claims to verify or certify test coverage.

✦ Documentation

Document

Why it matters

QA methodology

API questions, equivalence classes and story generation

Product contract

Scope and architecture

Case & evidence schema

Traceability and status semantics

Roadmap

Features and GitHub issue tracking

MCP distribution

Registry and free local server strategy

Release checklist

Release hygiene

Flowstead case study

Private real-world validation plan

Development:

python -m pip install -e '.[dev]'
python -m unittest discover -s tests -v
ruff check src tests

Provenance: All distributed code and examples in this repository were authored independently. The public package contains no reconstructed archived source, archived documentation, or private Flowstead fixtures.


Better questions → better test stories → better evidence.

GitHub · Issues · MIT License

Available Tools

4 tools
qa_audit_case_referencesC

Locate candidate references to indexed case IDs; never claim tests pass.

Inputs: plan_path (required), tests_path (required).

qa_capabilitiesB

Describe available operations and evidence status.

Inputs: no named parameters.

qa_compile_indexed_casesC

Compile reviewed questionnaire answers into stable, unverified test cases.

Inputs: answers_path (required).

qa_discover_api_surfaceB

Read a local OpenAPI JSON file and return an endpoint-by-endpoint QA questionnaire.

Inputs: openapi_path (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 4 tools

    Disambiguation5/5

    Each tool has a distinct role in the QA workflow: capabilities reporting, API surface discovery, case compilation, and reference auditing. There is no meaningful overlap that would cause misselection.

    Naming Consistency4/5

    Three tools follow a clear qa_verb_noun pattern (qa_discover_api_surface, qa_compile_indexed_cases, qa_audit_case_references). qa_capabilities is a minor deviation as a noun-only tool, but it remains readable and consistent enough.

    Tool Count5/5

    Four tools is well-scoped for a narrow QA toolkit focused on OpenAPI questionnaire discovery, compilation, and auditing. Each tool appears to earn its place without excess.

    Completeness3/5

    The set covers a discovery-to-compilation-to-audit flow, but it lacks obvious lifecycle operations such as listing or retrieving compiled cases and updating/deleting indexed cases. These gaps may force agents to work around the surface when managing cases.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      A
      maintenance
      ocal-first, read-only MCP server that audits repository test architecture across ten language ecosystems. It returns deterministic evidence-backed findings, blockers, rankings, and repository-native test plans without modifying source files.
      20
      601 npm
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      A stdio MCP server that caches Swagger/OpenAPI documents for multiple backend services and provides offline API search, retrieval, and request example generation. It only contacts remote servers when explicitly refreshing the cache.
      11
      5 npm
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Exposes 11 agent-native QA tools over stdio so coding agents can discover a project, analyze risk, select the smallest relevant test set, generate and dry-run tests, triage failures, propose tiered heals, analyze flakes, generate quality reports, and evaluate release readiness. Planning and healing tools stay read-only or propose-only, while execution defaults to dry-run, keeping agent actions safe and evidence-backed.
      MIT