qa-toolkit-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@qa-toolkit-mcp discover the API surface from openapi.json in this project"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
🧭 Evidence-Driven QA
From API surfaces to test stories to traceable evidence.
Discover · Question · Specify · Index · Audit
An open-source Python CLI, local MCP server and portable Agent Skill for systematic QA planning.
Quick start · The methodology · Agent setup · MCP setup · Documentation
✦ The idea
Great integration tests begin before writing test code. First map the actual API surface. For every operation ask: who may do this, who must be denied, what conditions and relationships must hold, what changes, and what observable result would prove it? Convert the answers into test stories, assign durable case IDs, and then inspect the existing test suite.
OPENAPI / SOURCE CONTRACT
│
▼
① DISCOVER TESTABLE SURFACE
endpoints · actors · states
│
▼
② ASK & ANSWER QUESTIONS
allowed? forbidden? conditions?
relationships? effects? proof?
│
▼
③ DESIGN TEST STORIES
positive · negative · edge cases
│
▼
④ CREATE INDEXED CASES
stable IDs · expected evidence
│
▼
⑤ AUDIT EXISTING TESTS
reference matching (not a PASS)
│
▼
⑥ IMPLEMENT → EXECUTE → REPORT
planned next stagesThe design is agent-assisted, deterministic at the tool boundary. AI agents reason about semantics and ask for missing product decisions. The Python core extracts the surface, validates reviewed stories, generates consistent case IDs and searches existing test sources. It never guesses an authorization policy from OpenAPI or treats a matching test ID as proof of a passing test.
Related MCP server: swagger-mcp
✦ Quick start
Install the package (Python 3.11+):
pip install evidence-driven-qaOr run without a permanent installation using uvx:
uvx --from evidence-driven-qa qa-toolkit --help1 · Discover the HTTP API surface from a saved local OpenAPI JSON file:
qa-toolkit surface --openapi openapi.json --json > qa-surface.jsonThis produces an editable per-endpoint questionnaire: declared routes, input parameters, response codes, available testing topics, and questions about intended behavior.
2 · Answer the contract questions and write test stories. Give qa-surface.json and the application's requirements to your agent. Ask it to complete the contract and stories fields using evidence from your project's specification, and to flag ambiguous requirements. Review its answers before compiling them.
3 · Create indexed test cases:
qa-toolkit cases --answers qa-surface.json > INDEXED_TEST_PLAN.mdOr use --json to produce a machine-readable registry. Generated IDs are deterministic for each method + path + story key and every new case starts as specified_unverified.
4 · Audit the tests that already exist:
qa-toolkit audit --plan INDEXED_TEST_PLAN.md --tests tests/ --jsonThe audit finds potential matches in pytest/Vitest/Playwright test source files. A reference is marked referenced_unverified, not covered or passed. Assertion-level review and executed test evidence are separate stages.
Try the independent synthetic example: example API contract, OpenAPI file, reviewed answers. These are fictitious fixtures, not production data.
qa-toolkit surface --openapi examples/synthetic-api/openapi.json
qa-toolkit cases --answers examples/synthetic-api/answers.json✦ How it works
The workflow is endpoint-by-endpoint, but not merely endpoint-by-endpoint testing. Resource relationships, states, and actors matter as much as HTTP routes.
Step | Questions the agent answers | Output |
Surface discovery | What endpoints, verbs, parameters, responses and security declarations exist? | OpenAPI inventory |
Intended behavior | What user/business capability should this operation provide? | Requirement-backed contract |
Allowed vs forbidden | Who is permitted? Who must be denied, including anonymous and foreign owners? | Actor/access matrix |
Conditions & relationships | Which resource IDs, ownership, tenant, parent/child relations and lifecycle states must hold? | Test-data dimensions |
Side effects & observability | What changes or stays unchanged? How do we prove that from HTTP and persistent state? | Concrete assertion targets |
Stories & case IDs | Which positive, forbidden, invalid, missing and transition scenarios expose a meaningful bug? | Stable indexed cases |
Audit & evidence | Which existing tests actually assert these outcomes and which have run? | References now; verified execution planned |
More than security testing
Test stories can target authentication and authorization, input validation, business rules, resource isolation, information disclosure, pagination and filtering, async queues and timeouts, idempotency and retries, concurrency, data integrity, error contracts, rate limiting, exports, compatibility, accessibility and failure recovery. Select applicable dimensions—not a mechanically generated Cartesian product of every possible input.
For example, a negative authorization story may require checking both the rejected status and that the response reveals nothing about a foreign record. For an asynchronous job, the required proof may include a real queued state followed by an eventual completion in a disposable environment. See the complete methodology.
✦ Agent setup
The skill supplies a reusable workflow for discovering contract gaps and writing meaningful, indexed test stories. It can work with any agent that supports the Agent Skills standard.
Install to supported coding agents:
npx skills add darizae/qa-toolkit --skill evidence-driven-qaOr install manually:
# Claude Code — user-level
mkdir -p ~/.claude/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md ~/.claude/skills/evidence-driven-qa/
# Codex / cross-agent project-level
mkdir -p .agents/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md .agents/skills/evidence-driven-qa/Suggested first agent instruction:
Use the evidence-driven-qa skill. Discover the testable API surface, inspect the requirements for each endpoint, answer who is allowed or denied and under which conditions, then design positive/negative/edge test stories. Produce stable case IDs and audit existing tests. Call out any unresolved business rules; do not assume implementation behavior is the intended contract.
Model and coding-agent ecosystem
The toolkit does not depend on a particular language model. Via compatible coding agents or MCP-capable hosts, it is suitable for workflows using these common model families:
Model family | Example agent/host |
OpenAI GPT models (GPT-6, GPT-5 series, GPT-4o) | Codex and other MCP-aware clients |
Anthropic Claude (Opus, Sonnet, Haiku) | Claude Code, Claude Desktop |
Google Gemini (Pro, Flash) | Gemini CLI and compatible agent hosts |
Meta Llama, Qwen, Mistral, DeepSeek | Local or hosted models through MCP-capable agent runtimes |
xAI Grok | MCP-capable agent integrations |
MCP integration is provided by the host or agent runtime, not by model weights themselves. The generic skill can also be read as instructions by agents without MCP support.
✦ MCP server setup
The Python distribution also provides qa-toolkit-mcp, a local, read-only stdio server using the same core as the CLI. No hosted account, API key, or commercial license is required.
Claude Code
claude mcp add --transport stdio qa-toolkit -- \
uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/projectCodex CLI
codex mcp add qa-toolkit -- \
uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/projectGeneric MCP stdio client configuration (replace the project path):
{
"mcpServers": {
"qa-toolkit": {
"command": "uvx",
"args": [
"--from", "evidence-driven-qa",
"qa-toolkit-mcp",
"--project", "/absolute/path/to/your/project"
]
}
}
}The MCP server exposes four read-only tools:
Tool | Purpose |
| List operations and current evidence limits |
| Load a local OpenAPI JSON file and return the questionnaire |
| Compile reviewed answers into indexed case records |
| Match case IDs against test-source references |
All paths are restricted to the selected project root. The MCP tools do not launch applications, run shell commands, modify files, access credentials, or connect to external services. The local stdio package is distinct from a hosted HTTPS ChatGPT app.
✦ Open-source by design
MIT-licensed. Free to use, study, modify, fork, redistribute and build upon—including commercial use under the terms of the license. The MCP server is part of the same open-source project; no subscription or paid account is needed.
Contributions welcome: bug reports, adapters, security improvements, documentation and reproducible examples. Open an issue before substantial API or schema changes.
This is an early release of the discovery and planning workflow. Test generation into project-native source, controlled test execution, browser evidence ingestion and HTML reports are on the roadmap. No current command claims to verify or certify test coverage.
✦ Documentation
Document | Why it matters |
API questions, equivalence classes and story generation | |
Scope and architecture | |
Traceability and status semantics | |
Features and GitHub issue tracking | |
Registry and free local server strategy | |
Release hygiene | |
Private real-world validation plan |
Development:
python -m pip install -e '.[dev]'
python -m unittest discover -s tests -v
ruff check src testsProvenance: All distributed code and examples in this repository were authored independently. The public package contains no reconstructed archived source, archived documentation, or private Flowstead fixtures.
Better questions → better test stories → better evidence.
GitHub · Issues · MIT License
Available Tools
4 toolsqa_audit_case_referencesC
Locate candidate references to indexed case IDs; never claim tests pass.
Inputs: plan_path (required), tests_path (required).
qa_capabilitiesB
Describe available operations and evidence status.
Inputs: no named parameters.
qa_compile_indexed_casesC
Compile reviewed questionnaire answers into stable, unverified test cases.
Inputs: answers_path (required).
qa_discover_api_surfaceB
Read a local OpenAPI JSON file and return an endpoint-by-endpoint QA questionnaire.
Inputs: openapi_path (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool has a distinct role in the QA workflow: capabilities reporting, API surface discovery, case compilation, and reference auditing. There is no meaningful overlap that would cause misselection.
Three tools follow a clear qa_verb_noun pattern (qa_discover_api_surface, qa_compile_indexed_cases, qa_audit_case_references). qa_capabilities is a minor deviation as a noun-only tool, but it remains readable and consistent enough.
Four tools is well-scoped for a narrow QA toolkit focused on OpenAPI questionnaire discovery, compilation, and auditing. Each tool appears to earn its place without excess.
The set covers a discovery-to-compilation-to-audit flow, but it lacks obvious lifecycle operations such as listing or retrieving compiled cases and updating/deleting indexed cases. These gaps may force agents to work around the surface when managing cases.
Maintenance
Related MCP Connectors
Read-only MCP server over the developer docs for the portfolio's JSON API, MCP, A2A and NLWeb.
Read-only MCP server for repository context: context packs, graph impact analysis, and validation.
Read-only MCP server that answers questions about turva.dev from its published data. Five tools return JSON: the service catalog with prices, contact and operator details, engagement principles and dated agent-readiness and security evidence with verification links. Connect over Streamable HTTP. You need no API key. The server does not scan other websites or run audits.
Evidence-readiness MCP server: validate, audit, and score briefs, memos, and evidence packs.
Related MCP Servers
- AlicenseAqualityAmaintenanceocal-first, read-only MCP server that audits repository test architecture across ten language ecosystems. It returns deterministic evidence-backed findings, blockers, rankings, and repository-native test plans without modifying source files.20601 npmMIT
- AlicenseAqualityDmaintenanceA stdio MCP server that caches Swagger/OpenAPI documents for multiple backend services and provides offline API search, retrieval, and request example generation. It only contacts remote servers when explicitly refreshing the cache.115 npm1MIT
- AlicenseNot gradedqualityCmaintenanceRead-only MCP server for Markdown, MDX, and OpenAPI docs, exposing list, search, fetch, and OpenAPI spec tools over stdio.1MIT
- AlicenseNot gradedqualityCmaintenanceExposes 11 agent-native QA tools over stdio so coding agents can discover a project, analyze risk, select the smallest relevant test set, generate and dry-run tests, triage failures, propose tiered heals, analyze flakes, generate quality reports, and evaluate release readiness. Planning and healing tools stay read-only or propose-only, while execution defaults to dry-run, keeping agent actions safe and evidence-backed.MIT