qai-consultant-mcp
by gvasile29
README.md
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/brand/qai_logo_horizontal_dark_1680.png">
<img alt="QAI Consultant" src="assets/brand/qai_logo_horizontal_1680.png" width="360">
</picture>
</p>
# QAI Consultant
An open-source AI agent that acts as a senior QA Architect — automatically generating a **Test Strategy**, **Risk Register**, **Effort Estimation Report**, and **Test Plan** from a simple project description, plus deterministic **QA Document Quality Review**, **Test Results Analysis**, and **QA Maturity Assessment** for evaluating what already exists. Also available as a local **MCP server** so Claude Code, Claude Desktop, and other MCP clients can ground their own QA work in the same standards and numbers.
> 🌐 **Live demo:** [quality-ai-consultant.streamlit.app](https://quality-ai-consultant.streamlit.app)
> 🔌 **MCP server:** `uvx qai-consultant-mcp` — no API keys, no Pinecone. See [MCP Server](#mcp-server-for-claude-code-claude-desktop-and-other-mcp-clients) below or the [package on PyPI](https://pypi.org/project/qai-consultant-mcp/).
> 🤖 Built with [Claude Code](https://claude.ai/code) by Anthropic.
---
## Screenshots
### Web UI (Streamlit)
**Landing page**
**MCP panel in the sidebar**
**Project Discovery dialogue** (Web Application template applied)
**Executive Readout + generation status**
**Risk Register** (Risk Ledger table)
**Effort Estimation**
**Test Strategy**
**Test Plan**
**QA Document Quality Review**
**QA Maturity Assessment**
### CLI
---
## Quick Start
### Option A — Use the live app (no setup)
👉 [quality-ai-consultant.streamlit.app](https://quality-ai-consultant.streamlit.app)
### Option B — Run locally
```bash
# 1. Clone and install
git clone https://github.com/gvasile29/qai-consultant.git
cd qai-consultant
pip install -r requirements.txt
# 2. Set up API keys
cp .env.example .env
# Edit .env and fill in the 4 keys (see Prerequisites below)
# 3. Build the knowledge base (one-time, pushes to Pinecone)
python src/ingest.py
# 4. Run
python src/cli.py # Terminal UI
streamlit run src/app.py # Web UI → http://localhost:8501
```
> 📖 Full installation guide: [INSTALL.md](INSTALL.md)
---
## The Problem
Creating a Test Strategy from scratch is time-consuming and requires deep QA expertise. Most teams either skip it, do it superficially, or spend days researching methodologies.
QAI Consultant eliminates this bottleneck by combining established QA methodologies, industry standards (ISTQB, OWASP, ISO 26262, A-SPICE), and expert knowledge into an AI agent that thinks like a seasoned QA Architect.
---
## Who Is This For?
- **QA Engineers** who need structured guidance on test strategy
- **Engineering Managers** who need effort estimations and resource planning
- **Development teams** without a dedicated QA Architect
- **QA Consultants** who want to accelerate their delivery
---
## What QAI Consultant Generates
From a single 11-question dialogue, QAI Consultant automatically generates **four documents**:
| Document | What it contains |
|---|---|
| ⚠️ **Risk Register** | Risk matrix, likelihood/impact analysis, mitigations per risk |
| 📊 **Effort Estimation Report** | PERT-based breakdown, team capacity analysis, confidence score |
| 📋 **Test Strategy** | ISTQB-aligned strategy tailored to your stack, methodology, and compliance |
| 📝 **Test Plan** | IEEE 829-aligned plan with entry/exit criteria, schedule, and AI tool oversight |
Above the four tabs, an **Executive Readout** summarizes overall risk, the QA effort range, team capacity, confidence, and the top 3 risks — computed deterministically from the Risk Register and Effort Estimation, with no extra LLM call.
All outputs are saved as Markdown files and available for PDF download.
**Evaluating what already exists** (deterministic, no LLM needed for the scores):
| Mode | What it does |
|---|---|
| 📝 **QA Document Quality Review** | Scores an existing Test Plan / Strategy / test case list 0–100 across six ISTQB/IEEE 829 dimensions, with findings and knowledge-base citations |
| 🧪 **Test Results Analysis** | Flaky, ever-failing, never-run and slowest tests plus failure clustering from JUnit XML/CSV — optionally used to ground the Risk Register in real execution data |
| 📈 **QA Maturity Assessment** | Indicative TMMi level (1–3) across 10 process areas from a free-text description, plus an EU AI Act Articles 9–15 readiness check for AI/ML systems |
---
## Knowledge Base
QAI Consultant's recommendations are grounded in real QA standards and methodologies:
- 📘 **ISTQB** — 14 certification syllabuses (CTFL, CTAL-TA, CTAL-TM, CTAL-TAE, CT-AI, and more)
- 🔒 **OWASP** — WSTG v4.2, MASTG, Top 10 2021
- 🚗 **ISO 26262** — Automotive functional safety (ASIL levels, HARA, V&V)
- 🏭 **A-SPICE** — Automotive SPICE process reference model (SWE.4, SWE.5, SWE.6)
- 📋 **IEEE 829** — Test documentation standard
- ⚙️ **ISO/IEC 25010** — Software product quality model
- 🇪🇺 **EU AI Act** — risk tiers, provider/deployer obligations, Article 50 transparency, Articles 9–15 testing implications
- 🧭 **Testing methodologies** — Agile, BDD/TDD, exploratory, risk-based testing, test pyramid
- 🔍 **Audit & process evaluation** — TMMi, CMMI, ISO/IEC 33002, ISO 19011, OWASP ASVS, ISO 27001, SOC 2, plus public failure case studies (Knight Capital, Boeing 737 MAX MCAS, CrowdStrike 2024)
- 🤖 **AI Test Planning** — 17 real-world AI SDLC case studies (2024–2025)
- 🧠 **Expert Knowledge** — Real QA scenarios and lessons learned from practitioners
---
## Prerequisites
QAI Consultant runs on cloud APIs — no local GPU required.
You need four API keys in a `.env` file (all have free tiers):
| Key | Where to get it |
|-----|----------------|
| `MISTRAL_API_KEY` | [console.mistral.ai](https://console.mistral.ai/) → API Keys (the default model, `ministral-14b-2512`, works on Mistral's free plan) |
| `OPENROUTER_API_KEY` | [openrouter.ai/keys](https://openrouter.ai/keys) (fallback; uses free models only) |
| `PINECONE_API_KEY` | [pinecone.io](https://www.pinecone.io/) → API Keys |
| `PINECONE_INDEX_NAME` | Name of your Pinecone index (e.g. `qai-consultant`, dimensions: 384, metric: cosine) |
```bash
cp .env.example .env
# Edit .env and fill in all four values
```
---
## Architecture
## How It Works
```
You describe your project (11 questions)
↓
QAI retrieves relevant knowledge from Pinecone (parallel RAG, 3 threads)
↓
QAI analyzes risks from your context → Risk Register (Mistral API)
↓
QAI estimates effort using PERT + industry benchmarks → Effort Report
↓
QAI summarizes risk, effort, capacity and top risks → Executive Readout (no LLM)
↓
QAI generates a Test Strategy backed by QA standards → Test Strategy (Mistral API)
↓
QAI generates an IEEE 829-aligned Test Plan → Test Plan (Mistral API)
↓
Four documents ready for Markdown + PDF download
```
LLM calls use the **Mistral API** (`ministral-14b-2512`) as the primary provider, with **OpenRouter** free models (Nemotron 3 Super → GLM 5.2) as automatic fallback.
---
## Interfaces
### Web UI (Browser — recommended)
```bash
streamlit run src/app.py
```
Or use the **live hosted version**: [quality-ai-consultant.streamlit.app](https://quality-ai-consultant.streamlit.app)
Besides the Test Strategy flow, the landing page and sidebar offer **Review an existing QA document** and **Assess QA Maturity**; test execution results can be attached on the review screen before generating.
### CLI (Terminal)
```bash
python src/cli.py # interactive Test Strategy flow
python src/cli.py --results run1.xml run2.xml # same flow, grounded in JUnit XML/CSV results
python src/cli.py --review path/to/test_plan.md # QA Document Quality Review
python src/cli.py --maturity path/to/process.txt # QA Maturity Assessment
```
### MCP Server (for Claude Code, Claude Desktop, and other MCP clients)
[](https://glama.ai/mcp/servers/gvasile29/qai-consultant)
[](https://glama.ai/mcp/servers/gvasile29/qai-consultant)
Listed on the [official MCP registry](https://registry.modelcontextprotocol.io) (`io.github.gvasile29/qai-consultant-mcp`), [Glama](https://glama.ai/mcp/servers/gvasile29/qai-consultant), and [Awesome MCP Servers](https://github.com/punkpeye/awesome-mcp-servers).
QAI Consultant is also available as a local, fully keyless MCP server —
`qai-consultant-mcp`. No Pinecone, no Mistral/OpenRouter API keys: it runs a
local embedding index over the knowledge base's Markdown documents (the ISTQB and
OWASP PDFs are not bundled, for licensing reasons) and exposes deterministic QA
effort estimation, document review, test-results analysis and maturity
assessment, so your own AI assistant can ground its QA work directly, with no
separate LLM call. It runs locally over stdio (requires [uv](https://docs.astral.sh/uv/));
a hosted version connectable from claude.ai is on the roadmap (v4.0).
```bash
uvx qai-consultant-mcp
```
**Claude Code:**
```bash
claude mcp add qai-consultant -- uvx qai-consultant-mcp
```
**Claude Desktop** (`claude_desktop_config.json`):
```json
{
"mcpServers": {
"qai-consultant": {
"command": "uvx",
"args": ["qai-consultant-mcp"]
}
}
}
```
**Tools:**
| Tool | What it does |
|---|---|
| `retrieve_qa_knowledge` | Grounding chunks from the KB (standards summaries — OWASP Top 10, IEEE 829, ISO/IEC 25010, ISO 26262, A-SPICE, EU AI Act — plus methodologies, audit/evaluation guides, and case studies), filterable by category |
| `list_kb_sources` | Every document in the KB, grouped by category |
| `estimate_qa_effort` | Deterministic PERT-based effort estimate (no LLM narrative — you write your own from the numbers) |
| `review_qa_document` | Deterministic 0–100 quality score for an existing Test Plan/Strategy/test case list across six ISTQB/IEEE-829 dimensions, with findings + KB citations |
| `analyze_test_results` | Deterministic health metrics from JUnit XML/CSV test execution data — flaky tests, ever-failing tests, slowest tests, failure clustering |
| `assess_qa_maturity` | Deterministic indicative TMMi process-maturity level (1-3, never a certified 4-5) from a free-text description, plus a conditional EU AI Act Articles 9-15 readiness score when the input signals an AI/ML system |
**Prompts:** `qa_project_interview` (the same 11-question intake this app uses), `risk_register_structure`, `test_strategy_structure`, `test_plan_structure` — each grounds the client's generation in `retrieve_qa_knowledge` with `[Source N]` citations.
**Privacy:** anonymous usage telemetry is on by default (since v3.7.0); turn it off with `QAI_TELEMETRY=0` or `DO_NOT_TRACK=1`. Only tool name/success/duration/category, version and OS family are sent, under a random ID that's new for every server run and never stored — never your query text or project details.
---
## Feedback Loop
After each generation, QAI asks: *"Was this strategy useful?"*
- **Yes** → strategy saved to `knowledge_base/generated_strategies/` and included in the next re-ingestion
- **Partially** → strategy saved with your improvement notes
- **No** → discarded
This creates a **feedback loop** where QAI learns from validated real-world outputs over time.
---
## Roadmap
- **v0.1** ✅ Core agent + CLI + Streamlit Web UI
- **v0.2** ✅ Feedback loop — validated strategies grow the knowledge base
- **v0.3** ✅ Risk Register — automatic risk analysis alongside Test Strategy
- **v0.4** ✅ Effort Estimation Report — PERT-based with team capacity analysis
- **v0.5** ✅ Auto re-ingest — file watcher + incremental ingest + manifest tracking (the file watcher was removed in v2.0)
- **v0.6** ✅ Confidence level algorithm — score-based (0-100): PERT spread + capacity gap + data quality + multiplier magnitude
- **v1.0** ✅ MVP — error handling, input validation, logging, full documentation, tests, Apache 2.0 license
- **v2.0** ✅ Cloud migration — Ollama → Mistral API + OpenRouter fallback; ChromaDB → Pinecone; deployed to Streamlit Cloud
- **v2.0.1** ✅ Stability — 27 bugs fixed: PERT normalization, template application, PDF caching, session state, filename sanitization, RAG fallback, per-step exception isolation
- **v2.0.2** ✅ Stability — release-gate evals (estimate integrity + RAG metrics), 5 estimation/validation defects fixed, session-state crash fix, narrative duplication/truncation fixes, per-step generation isolation from LLM outages
- **v2.5.0** ✅ In-app Release Notes — sidebar panel + one-time "what's new" banner
- **v2.5.1** ✅ Knowledge base — new `evaluation_audit/` pillar: process/test maturity models, audit methodology, security/compliance audit, real public failure case studies
- **v2.5.2** ✅ EU AI Act Article 50 transparency patch — sidebar AI-interaction notice + visible "AI-generated content" label on every generated document
- **v2.6.0** ✅ EU AI Act knowledge base pillar — risk tiers, provider/deployer obligations, Article 50 transparency, Articles 9-15 testing implications, conformity assessment, timeline
- **v3.0.0** ✅ MCP server MVP — local, keyless `qai-consultant-mcp` (standards-grounded retrieval + deterministic effort estimation), in-app announcement, and machine-readable AI-generated marking (EU AI Act Article 50(2))
- **v3.1.0** ✅ Evaluation Package — QA Document Quality Review (deterministic ISTQB/IEEE-829 rubric scoring an existing Test Plan/Strategy/test case list, with an optional AI narrative) and Test Results Analysis (flaky/ever-failing/slowest/failure-clustering metrics from JUnit XML/CSV, grounding the Risk Register in real execution data); available in the web app, CLI (`--review`, `--results`), and the MCP server (`review_qa_document`, `analyze_test_results`)
- **v3.1.1** ✅ Visit counter — a running total of app visits now shows in the sidebar, persisted in Pinecone so it survives redeploys
- **v3.1.2** ✅ Fix — the 3.1.1 visit counter never actually incremented (Pinecone rejected its all-zero placeholder vector); now works correctly
- **v3.1.3** ✅ Fix — the visit counter's label was in Romanian ("vizite") instead of English; now reads "visits" to match the rest of the app's UI copy
- **v3.1.4** ✅ Added the `mcp-name` marker to `README_MCP.md` (PyPI long description) — a prerequisite for listing `qai-consultant-mcp` in the official Anthropic MCP registry; no functional change
- **v3.1.5** ✅ Fix — `qai-consultant-mcp` failed to start (`ModuleNotFoundError: mcp.server.fastmcp`) after the upstream `mcp` SDK's breaking 2.0.0 release removed the `FastMCP` module the server depends on; `mcp` is now pinned to `>=1.8.0,<2.0.0`
- **v3.1.6** ✅ Fix — `qai-consultant-mcp` could fail to attach in Claude Desktop on a cold cache (a client-side handshake timeout, since the server used to fully embed the whole knowledge base before responding to `initialize`); the full index build is now lazy, deferred until the first real request
- **v3.2** ✅ CI quality gates completion — a separate nightly workflow exercising real Pinecone/Mistral/OpenRouter contract tests, which never blocks a PR
- **v3.3** ✅ Adopted the EU's official AI-generated-content icon (Code of Practice, AI Act Article 50(4)) in the Streamlit sidebar and all generated-document PDF exports, reinforcing the existing text/metadata disclosure
- **v3.3.1** ✅ Fix — `qai-consultant-mcp` could intermittently fail to attach in Claude Desktop because 4 of its 6 runtime dependencies had loose version bounds, letting `uv` re-resolve and reinstall on any unrelated upstream release; all dependencies are now exact-pinned
- **v3.4** ✅ App visual redesign ("Calibration Bench") — token-based color/typography system (IBM Plex fonts, no font CDN) and a reusable "Signal Ledger" score/severity component, applied to Document Review, Effort confidence, Results Analysis, Risk Register, and the Project Discovery question list
- **v3.4.1** ✅ Distribution links (official MCP registry, Glama, Awesome MCP Servers) now shown in the app's own MCP panel, not just the READMEs; landing page redesign started (Phase 1 of 3 — see the CHANGELOG)
- **v3.4.2** ✅ Interactive flow redesign (Phase 2 of 3 — see the CHANGELOG): the Project Discovery dialogue, review screen, and sidebar now share the landing page's "Power-On Sequence" visual language (animated progress bar, one-time entrance-animated summary tiles, hover feedback)
- **v3.4.3** ✅ Output screens redesign (Phase 3 of 3, completing the "Power-On Sequence" redesign — see the CHANGELOG): the Test Strategy results view and the document review screen now share the same visual language (live 4-stage sequence status, entrance-animated score tiles, hover feedback), plus a "What you get in ~2 minutes" addendum on the landing screen
- **v3.4.4** ✅ Fix — `qai-consultant-mcp` could again fail to attach in Claude Desktop, this time from an *unpinned transitive* dependency (scipy, via scikit-learn) picking up a fresh release mid-cache-miss; the entire resolved dependency tree (~99 entries) is now exact-pinned, not just the 6 direct imports
- **v3.5.0** ✅ QA Maturity Assessment (`assess_qa_maturity`) — deterministic TMMi process-maturity signal (10 process areas, indicative level 1-3) plus a conditional EU AI Act Articles 9-15 readiness score for AI/ML projects; available in the web app ("📈 Assess QA Maturity"), CLI (`--maturity`), and the MCP server (`assess_qa_maturity`)
- **v3.5.1** Fix: QA Maturity scoring missed paraphrased evidence (requirement traceability described in prose, progress tracking described as "defect triage"/"pass rate") — found via live browser QA. **Yanked from PyPI** — see v3.5.2, published the same day, for why
- **v3.5.2** ✅ Fix: the v3.5.1 MCP package could not be installed at all (`torch==2.13.0+cpu` unresolvable from plain PyPI) — reverted to a plain `torch==2.13.0` pin. This is the first working PyPI publish of the `assess_qa_maturity` MCP tool, since v3.5.0's own publish step was never completed and v3.5.1 was broken
- **v3.5.3** ✅ `qai-consultant-mcp`'s local index switched from `sentence-transformers`/`torch` to `fastembed` (ONNX Runtime) for embeddings — same retrieval quality (`evals/local_index_parity.py`: recall@5=0.91, MRR=0.86, unchanged), ~3x faster cold import, and a much smaller dependency list. Removed the weekly dependency-drift-canary workflow, no longer justified at the smaller scale. See `CHANGELOG.md` for details.
- **v3.5.4** ✅ Reliability fixes from the 2026-09-23 external audit: streaming no longer duplicates output when the primary LLM drops mid-response; the effort estimate's risk buffer now counts actual Risk Matrix rows (it was inflated to the 35% cap by keyword matches); confidence scoring no longer treats precise answers like "None" or "functional safety" as vague. See `CHANGELOG.md`.
- **v3.6.0** ✅ Public-app protection and an at-a-glance summary, from the 2026-09-23 external audit: server-side daily generation limits (a global daily cap plus a per-visitor cap, alongside the existing 3-runs-per-session cap) so opening a new tab no longer resets the quota; and an **Executive Readout** above the four output tabs — overall risk, QA effort range, team capacity, confidence, and the top 3 risks to address first — built deterministically from the Risk Register and Effort Estimation, with no extra LLM call. See `CHANGELOG.md`.
- **v3.6.1** ✅ The OpenRouter fallback now uses only free-tier models (Nemotron 3 Super → GLM 5.2, via OpenRouter's `models` fallback chain), so the fallback no longer accrues charges; an empty AI response now shows a clear error instead of an empty document; the in-app AI notice now warns that submitted text may be logged and used for training by the LLM providers. See `CHANGELOG.md`.
- **v3.6.2** ✅ Primary model switched to Ministral 14B (Mistral Small is rate-limited on Mistral's free plan), output budget raised to 6,500 tokens so documents are no longer cut off, and a bold Risk Matrix header no longer empties the Risk Ledger, Executive Readout and effort risk buffer
- **v3.6.3** ✅ Fixed the MCP knowledge-base tools crashing with a numpy "inhomogeneous shape" error for installs that cached a briefly broken upstream copy of the embedding model (fastembed 0.8.1) See `CHANGELOG.md`.
- **v3.7.1** ✅ MCP telemetry never sends an unrecognized `category` value (it could carry free text), `analyze_test_results` docs now match how it handles tests with fewer than 3 runs, and a new agent skill (`skills/qai-consultant/`) tells AI agents when to call each MCP tool. See `CHANGELOG.md`.
- **v3.7.0** ✅ Live adoption numbers on the landing page (visits, AI runs, MCP installs, GitHub stars; downloads and usefulness once there's data), download + feedback counters, and MCP telemetry on by default with an opt-out. See `CHANGELOG.md`.
- **v4.0** Remote MCP — a hosted server connectable from claude.ai, plus server-side usage metrics
---
## Contributing
QAI Consultant is built by the QA community, for the QA community.
Contributions are welcome:
- 📚 Add new knowledge sources to `knowledge_base/`
- 🧠 Share expert knowledge using the prompts in `knowledge_base/expert_knowledge/`
- 🐛 Report bugs or suggest features via GitHub Issues
- 🔧 Submit pull requests
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
---
## Troubleshooting
| Problem | Solution |
|---|---|
| "Missing required secret: 'MISTRAL_API_KEY'" | Add your key to `.env` or Streamlit Cloud secrets |
| "Missing required secret: 'PINECONE_API_KEY'" | Add your Pinecone key to `.env` |
| "Knowledge base is empty" | Run `python src/ingest.py` to push documents to Pinecone |
| "The AI providers are temporarily unavailable" | Both Mistral and the OpenRouter free models failed — retry in a few minutes; `logs/qai_consultant.log` has the underlying error |
| Mistral returns `429 Rate limit exceeded` (code 1300) at almost no usage | On Mistral's free plan, Mistral Small/Medium are rejected while the Ministral models are served — keep the default `ministral-14b-2512`, or enable pay-as-you-go before switching `MISTRAL_MODEL` |
> 📖 Full troubleshooting guide: [INSTALL.md](INSTALL.md#troubleshooting)
TDQS
A4.4/5.0
Scored across 6 tools
Disambiguation5/5
Each tool has a clearly distinct purpose: knowledge retrieval, source listing, effort estimation, document review, maturity assessment, and test results analysis. No two tools overlap in functionality; an agent can unambiguously select the correct tool for a task.
Naming Consistency5/5
All tool names follow a consistent verb_noun pattern (retrieve_qa_knowledge, list_kb_sources, estimate_qa_effort, review_qa_document, assess_qa_maturity, analyze_test_results). The naming is uniform, predictable, and clearly indicates the action and object.
Tool Count5/5
Six tools is well within the typical 3-15 range and each tool serves a distinct, non-redundant function in the QA consulting domain. The scope is tight without being sparse or overwhelming.
Completeness4/5
The tool set covers the core lifecycle: knowledge retrieval, source enumeration, effort estimation, document review, maturity assessment, and test result analysis. Minor gaps exist (e.g., no tool for generating a QA plan or updating knowledge), but these are not essential to the server's stated consultative purpose and agents can work around them.
Maintenance
ActivityActive
ResponsivenessSlow