AI Agent Release Assurance MCP
This MCP server provides read-only release-assurance tools for assessing release readiness, test failures, defect hotspots, and regression plans.
Assess a release’s readiness and get an explainable GO, CONDITIONAL_GO, or NO_GO recommendation with risk evidence.
Retrieve failed and blocked tests for a release, optionally filtered by criticality.
Rank release components by severity-weighted unresolved defect risk.
Build a bounded, risk-based regression test plan using test and defect evidence.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@AI Agent Release Assurance MCP Assess release readiness for REL-2026.08.1"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
AI Agent Release Assurance MCP
Version 0.1: Release-intelligence foundation using synthetic QA data.
AI-agent evaluation capabilities are planned for Version 0.2.
An explainable Model Context Protocol (MCP) server that helps AI clients analyze software test results and defects and produce evidence-based release-readiness recommendations.
All releases, tests, defects, and customer-impact scenarios in this repository are fictional. No employer, customer, production, personal, or regulated data is used.
Why this project exists
Release decisions often require evidence distributed across test results, defect records, and team documentation.
This server gives an AI client a small, read-only interface for answering questions such as:
Should a particular release ship?
Which failed tests are potential release blockers?
Where is unresolved defect risk concentrated?
Which tests should be prioritized during targeted regression testing?
The AI does not invent the risk score. The server calculates it deterministically and returns the underlying evidence, weights, blockers, and recommended next actions for human review.
Related MCP server: QA Copilot AI
Current capabilities
Type | Name | Purpose |
Tool |
| Returns an explainable |
Tool |
| Retrieves failed and blocked tests with an optional criticality filter |
Tool |
| Ranks components by severity-weighted unresolved defect risk |
Tool |
| Creates a bounded, risk-based regression plan |
Resource |
| Lists the synthetic releases available for analysis |
Prompt |
| Guides an evidence-based release-readiness review |
Architecture
flowchart TD
A[AI host or MCP Inspector] -->|MCP request| B[Python MCP server]
B --> C[QA service and risk rules]
C --> D[(Synthetic SQLite data)]
D --> C
C -->|Structured evidence| B
B -->|Tool result| A
D -. optional migration .-> E[(Snowflake)]SQLite keeps Version 0.1 reproducible and credential-free. The optional snowflake/setup.sql file demonstrates a possible Snowflake-native MCP path.
Quick start
Requirements
Python 3.10 or newer
Node.js/npm for the visual MCP Inspector
Install and run
git clone https://github.com/Zoya-Ammar/ai-agent-release-assurance-mcp.git
cd ai-agent-release-assurance-mcp
uv sync --extra dev
uv run python -m banking_qa_mcp.seed
uv run mcp dev src/banking_qa_mcp/server.pyThe final command starts MCP Inspector.
Open Tools, select assess_release_readiness, and provide:
{
"release_id": "REL-2026.08.1"
}Expected headline result:
{
"recommendation": "NO_GO",
"risk_score": 100,
"test_pass_rate_percent": 62.5,
"blockers": [
"Open SEV1 defect",
"Failed or blocked critical test",
"Failed or blocked high-criticality test"
]
}For comparison, REL-2026.08.2 returns GO with a risk score of 7.
Run the tests
Run the full automated test suite:
uv run pytest -qRun the dependency-free core verification:
uv run python scripts/smoke_test.pyVersion 0.1 includes tests for:
High-risk and lower-risk release recommendations
Test-result filtering
Regression-plan limits and prioritization
Invalid release identifiers
Explainable risk scoring
The score is capped at 100:
25 × failed or blocked critical tests
12 × failed or blocked high-criticality tests
35 × open SEV1 defects
18 × open SEV2 defects
7 × open SEV3 defects
2 × open SEV4 defectsAn open SEV1 defect, a failed or blocked critical test, or a failed or blocked high-criticality test is also reported as an explicit release blocker.
These weights are demonstration policy—not a universal financial-services or software-quality standard. In production, thresholds would require approval, version control, validation, and periodic review by the appropriate risk owners.
Security considerations
Version 0.1 is deliberately read-only at the application layer. A production implementation should also include:
Authentication and role-based authorization
Least-privilege database and service roles
Input and output validation
Audit logs for tool calls and recommendations
Rate limiting and observability
Secrets management and encrypted transport
Human approval for release decisions
Prompt-injection testing for retrieved content
The optional Snowflake example includes a native SQL-execution tool for sandbox demonstration purposes. It should be restricted through a dedicated read-only role and narrowed further before any non-demo use.
Version 0.2 roadmap
The next version will expand this release-intelligence foundation into an AI-agent assurance system.
Planned capabilities include:
An original AI-agent evaluation corpus
Grounding and citation validation
Prompt-injection resistance testing
Privacy and data-minimization checks
Accessibility and negative-path scenarios
Baseline-versus-candidate comparisons
Regression detection between agent versions
Playwright-based UI and accessibility execution
Snowflake-backed evaluation evidence
Human-reviewed AI-agent release recommendations
Project status
This repository is an educational portfolio prototype. It is not a production banking system, compliance tool, or autonomous release authority.
References
License
This project is available under the MIT License.
Version 0.2: AI-agent grounding evaluation (implemented)
Version 0.2 adds a limited, deterministic grounding/citation check while retaining all Version 0.1 release-intelligence tools. This is an educational first step toward the broader AI-agent assurance roadmap; security, privacy, accessibility, semantic entailment, Playwright execution, and agent-version comparisons are not yet implemented.
New MCP tools and resource
assess_agent_grounding(case_id): evaluate one synthetic agent answer's atomic claimsrun_grounding_evaluation_suite(): evaluate all bundled scenariosqa://grounding-cases: list available synthetic evaluation case IDs
Run the original setup commands, then open MCP Inspector and call
assess_agent_grounding with {"case_id":"AGENT-001"}. Try AGENT-002 through
AGENT-006 to inspect unsupported claims, missing citations, invented source IDs,
mixed outcomes, and normalization behavior. The bundled synthetic corpus is in
data/agent_evaluations.json and the implementation is in src/banking_qa_mcp/grounding.py.
Run uv run pytest -q to execute both existing release tests and new grounding tests.
Alternatively, with pytest installed, use PYTHONPATH=src python -m pytest -q.
How this evaluator works (and what it does not prove)
Each case supplies atomic claims, source text, and explicit citation IDs. For each claim the validator verifies that cited IDs exist and checks whether the entire normalized claim is a contiguous substring of a cited source. A case passes only when all claims have such evidence. It returns claim-level status and exact-support metrics.
This is not an LLM judge or semantic fact checker. Paraphrases may fail, and a copied false statement may pass. Claims are supplied explicitly rather than extracted from free-form agent answers. Therefore a PASS means exact textual support was located in provided cited text, not that an answer is factually correct, safe, or production ready. Do not use this score as a real-world deployment gate without more rigorous evaluation.
Available Tools
4 toolsassess_release_readinessB
Calculate an explainable GO, CONDITIONAL_GO, or NO_GO recommendation.
Inputs: release_id (required).
find_defect_hotspotsB
Rank release components by the weighted risk of unresolved defects.
Inputs: release_id (required). Returns: result
get_failed_testsB
Return failed and blocked tests, optionally filtered by criticality.
Inputs: release_id (required), criticality. Returns: result
recommend_regression_testsC
Build a risk-based regression plan grounded in test and defect evidence.
Inputs: release_id (required), max_tests.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool produces a distinct output: a GO/NO-GO decision, a filtered list of test failures, a component risk ranking, and a regression test plan. find_defect_hotspots and recommend_regression_tests share an evidence base of defect/test risk, but their purposes are clearly separated by output type, so misselection is unlikely.
All four tools follow a consistent verb_noun snake_case pattern (assess_release_readiness, get_failed_tests, find_defect_hotspots, recommend_regression_tests). The verb clearly signals the action (assess, get, find, recommend) and the noun signals the resource, making the pattern highly predictable.
Four tools is on the lean side but well-scoped for release assurance: each tool fills a distinct role covering evidence gathering, risk analysis, planning, and final decision. There is no redundancy or bloat, and every tool earns its place in the pipeline.
The set forms a coherent end-to-end release readiness workflow: pull test failures, rank defect hotspots, build a regression plan from that evidence, and produce a final GO/NO-GO assessment. Minor gaps exist, such as no tool to drill into individual defect details or fetch component/change scope, but agents can work around these.
Related MCP Connectors
Read-only AI coding tools for change verification, release readiness, capacity, and guidance.
QA platform for agents: coverage signals, in-repo test plans, verified tests and release governance.
Test analytics for AI agents: test history, flaky tests, failure clusters, coverage.
Diagnose AI workflows for failure, security, and handoff risks — RED/AMBER/GREEN per node.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables intelligent analysis of regression test failures and automatic discovery of solutions in JIRA. Analyzes test logs using AI-driven algorithms and matches errors with relevant JIRA issues through natural language interactions.-
- AlicenseBqualityDmaintenanceMCP server for AI-powered QA analysis. It enables analyzing test failures, identifying root causes, suggesting fixes, classifying defects, detecting flaky tests, and generating test cases and bug reports.10ISC
- AlicenseNot gradedqualityCmaintenanceEnables evidence-first release readiness assessment by running or accepting build, API, browser, visual, performance, and security evidence, then returning SHIP, REVIEW, or HOLD recommendations with clustered regressions.Apache 2.0
- AlicenseNot gradedqualityBmaintenanceEnables evaluating AI applications, inspecting reliability evidence, and gating releases from development and CI workflows.33Apache 2.0