Skip to main content
Glama
AI Agent Release Assurance MCP

AI Agent Release Assurance MCP

by Zoya-Ammar

AI Agent Release Assurance MCP

Version 0.1: Release-intelligence foundation using synthetic QA data.
AI-agent evaluation capabilities are planned for Version 0.2.

An explainable Model Context Protocol (MCP) server that helps AI clients analyze software test results and defects and produce evidence-based release-readiness recommendations.

All releases, tests, defects, and customer-impact scenarios in this repository are fictional. No employer, customer, production, personal, or regulated data is used.

Why this project exists

Release decisions often require evidence distributed across test results, defect records, and team documentation.

This server gives an AI client a small, read-only interface for answering questions such as:

  • Should a particular release ship?

  • Which failed tests are potential release blockers?

  • Where is unresolved defect risk concentrated?

  • Which tests should be prioritized during targeted regression testing?

The AI does not invent the risk score. The server calculates it deterministically and returns the underlying evidence, weights, blockers, and recommended next actions for human review.

Related MCP server: QA Copilot AI

Current capabilities

Type

Name

Purpose

Tool

assess_release_readiness

Returns an explainable GO, CONDITIONAL_GO, or NO_GO recommendation

Tool

get_failed_tests

Retrieves failed and blocked tests with an optional criticality filter

Tool

find_defect_hotspots

Ranks components by severity-weighted unresolved defect risk

Tool

recommend_regression_tests

Creates a bounded, risk-based regression plan

Resource

qa://releases

Lists the synthetic releases available for analysis

Prompt

release_go_no_go

Guides an evidence-based release-readiness review

Architecture

flowchart TD
    A[AI host or MCP Inspector] -->|MCP request| B[Python MCP server]
    B --> C[QA service and risk rules]
    C --> D[(Synthetic SQLite data)]
    D --> C
    C -->|Structured evidence| B
    B -->|Tool result| A
    D -. optional migration .-> E[(Snowflake)]

SQLite keeps Version 0.1 reproducible and credential-free. The optional snowflake/setup.sql file demonstrates a possible Snowflake-native MCP path.

Quick start

Requirements

  • Python 3.10 or newer

  • uv

  • Node.js/npm for the visual MCP Inspector

Install and run

git clone https://github.com/Zoya-Ammar/ai-agent-release-assurance-mcp.git
cd ai-agent-release-assurance-mcp
uv sync --extra dev
uv run python -m banking_qa_mcp.seed
uv run mcp dev src/banking_qa_mcp/server.py

The final command starts MCP Inspector.

Open Tools, select assess_release_readiness, and provide:

{
  "release_id": "REL-2026.08.1"
}

Expected headline result:

{
  "recommendation": "NO_GO",
  "risk_score": 100,
  "test_pass_rate_percent": 62.5,
  "blockers": [
    "Open SEV1 defect",
    "Failed or blocked critical test",
    "Failed or blocked high-criticality test"
  ]
}

For comparison, REL-2026.08.2 returns GO with a risk score of 7.

Run the tests

Run the full automated test suite:

uv run pytest -q

Run the dependency-free core verification:

uv run python scripts/smoke_test.py

Version 0.1 includes tests for:

  • High-risk and lower-risk release recommendations

  • Test-result filtering

  • Regression-plan limits and prioritization

  • Invalid release identifiers

Explainable risk scoring

The score is capped at 100:

25 × failed or blocked critical tests
12 × failed or blocked high-criticality tests
35 × open SEV1 defects
18 × open SEV2 defects
 7 × open SEV3 defects
 2 × open SEV4 defects

An open SEV1 defect, a failed or blocked critical test, or a failed or blocked high-criticality test is also reported as an explicit release blocker.

These weights are demonstration policy—not a universal financial-services or software-quality standard. In production, thresholds would require approval, version control, validation, and periodic review by the appropriate risk owners.

Security considerations

Version 0.1 is deliberately read-only at the application layer. A production implementation should also include:

  • Authentication and role-based authorization

  • Least-privilege database and service roles

  • Input and output validation

  • Audit logs for tool calls and recommendations

  • Rate limiting and observability

  • Secrets management and encrypted transport

  • Human approval for release decisions

  • Prompt-injection testing for retrieved content

The optional Snowflake example includes a native SQL-execution tool for sandbox demonstration purposes. It should be restricted through a dedicated read-only role and narrowed further before any non-demo use.

Version 0.2 roadmap

The next version will expand this release-intelligence foundation into an AI-agent assurance system.

Planned capabilities include:

  • An original AI-agent evaluation corpus

  • Grounding and citation validation

  • Prompt-injection resistance testing

  • Privacy and data-minimization checks

  • Accessibility and negative-path scenarios

  • Baseline-versus-candidate comparisons

  • Regression detection between agent versions

  • Playwright-based UI and accessibility execution

  • Snowflake-backed evaluation evidence

  • Human-reviewed AI-agent release recommendations

Project status

This repository is an educational portfolio prototype. It is not a production banking system, compliance tool, or autonomous release authority.

References

License

This project is available under the MIT License.

Version 0.2: AI-agent grounding evaluation (implemented)

Version 0.2 adds a limited, deterministic grounding/citation check while retaining all Version 0.1 release-intelligence tools. This is an educational first step toward the broader AI-agent assurance roadmap; security, privacy, accessibility, semantic entailment, Playwright execution, and agent-version comparisons are not yet implemented.

New MCP tools and resource

  • assess_agent_grounding(case_id): evaluate one synthetic agent answer's atomic claims

  • run_grounding_evaluation_suite(): evaluate all bundled scenarios

  • qa://grounding-cases: list available synthetic evaluation case IDs

Run the original setup commands, then open MCP Inspector and call assess_agent_grounding with {"case_id":"AGENT-001"}. Try AGENT-002 through AGENT-006 to inspect unsupported claims, missing citations, invented source IDs, mixed outcomes, and normalization behavior. The bundled synthetic corpus is in data/agent_evaluations.json and the implementation is in src/banking_qa_mcp/grounding.py.

Run uv run pytest -q to execute both existing release tests and new grounding tests. Alternatively, with pytest installed, use PYTHONPATH=src python -m pytest -q.

How this evaluator works (and what it does not prove)

Each case supplies atomic claims, source text, and explicit citation IDs. For each claim the validator verifies that cited IDs exist and checks whether the entire normalized claim is a contiguous substring of a cited source. A case passes only when all claims have such evidence. It returns claim-level status and exact-support metrics.

This is not an LLM judge or semantic fact checker. Paraphrases may fail, and a copied false statement may pass. Claims are supplied explicitly rather than extracted from free-form agent answers. Therefore a PASS means exact textual support was located in provided cited text, not that an answer is factually correct, safe, or production ready. Do not use this score as a real-world deployment gate without more rigorous evaluation.

Available Tools

4 tools
assess_release_readinessB

Calculate an explainable GO, CONDITIONAL_GO, or NO_GO recommendation.

Inputs: release_id (required).

find_defect_hotspotsB

Rank release components by the weighted risk of unresolved defects.

Inputs: release_id (required). Returns: result

get_failed_testsB

Return failed and blocked tests, optionally filtered by criticality.

Inputs: release_id (required), criticality. Returns: result

recommend_regression_testsC

Build a risk-based regression plan grounded in test and defect evidence.

Inputs: release_id (required), max_tests.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 4 tools

    Disambiguation4/5

    Each tool produces a distinct output: a GO/NO-GO decision, a filtered list of test failures, a component risk ranking, and a regression test plan. find_defect_hotspots and recommend_regression_tests share an evidence base of defect/test risk, but their purposes are clearly separated by output type, so misselection is unlikely.

    Naming Consistency5/5

    All four tools follow a consistent verb_noun snake_case pattern (assess_release_readiness, get_failed_tests, find_defect_hotspots, recommend_regression_tests). The verb clearly signals the action (assess, get, find, recommend) and the noun signals the resource, making the pattern highly predictable.

    Tool Count5/5

    Four tools is on the lean side but well-scoped for release assurance: each tool fills a distinct role covering evidence gathering, risk analysis, planning, and final decision. There is no redundancy or bloat, and every tool earns its place in the pipeline.

    Completeness4/5

    The set forms a coherent end-to-end release readiness workflow: pull test failures, rank defect hotspots, build a regression plan from that evidence, and produce a final GO/NO-GO assessment. Minor gaps exist, such as no tool to drill into individual defect details or fetch component/change scope, but agents can work around these.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      MCP server for AI-powered QA analysis. It enables analyzing test failures, identifying root causes, suggesting fixes, classifying defects, detecting flaky tests, and generating test cases and bug reports.
      10
      ISC
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables evidence-first release readiness assessment by running or accepting build, API, browser, visual, performance, and security evidence, then returning SHIP, REVIEW, or HOLD recommendations with clustered regressions.
      Apache 2.0