Skip to main content
Glama

Enterprise AI Agent Evaluation & Deployment Platform

A dependency-light evaluation platform for RAG/wiki-quality AI agents. It scores agent outputs for faithfulness, retrieval relevance, hallucination risk, latency, and cost; produces CI-friendly quality gates; emits regression/canary reports; and exposes the workflow through a lightweight MCP-style stdio tool server.

What Is Included

  • JSONL evaluation case format for RAG/wiki workflows.

  • Deterministic checks for:

    • faithfulness to retrieved context and reference answer,

    • retrieval relevance against question and expected keywords,

    • hallucination risk from unsupported answer content,

    • latency and cost thresholds.

  • 100+ synthetic case generator.

  • CI/CD-style suite-level and case-level quality gates.

  • Markdown and JSON evaluation reports.

  • Regression report comparing baseline and candidate runs.

  • Canary promotion policy with traffic ramp decisions.

  • OpenTelemetry-compatible JSONL traces/metrics.

  • MCP-style stdio server exposing evaluation tools.

Related MCP server: mcp-llm-eval

Quick Start

git clone https://github.com/ad-github1/ENTERPRISE-AI-AGENT-EVALUATION-PLATFORM.git
cd ENTERPRISE-AI-AGENT-EVALUATION-PLATFORM
PYTHONPATH=src python3 -m agent_eval_platform generate-cases --count 120 --out examples/wiki_eval_cases.jsonl
PYTHONPATH=src python3 -m agent_eval_platform evaluate \
  --cases examples/wiki_eval_cases.jsonl \
  --gate examples/quality_gate.json \
  --variant candidate \
  --json-out reports/eval_result.json \
  --markdown-out reports/eval_report.md \
  --traces-out reports/traces.jsonl
PYTHONPATH=src python3 -m unittest discover -s tests

Testing

Run the test suite:

PYTHONPATH=src python3 -m unittest discover -s tests

Current local result:

Ran 5 tests in 0.015s

OK

After installation:

pip install -e .
agent-eval evaluate --cases examples/wiki_eval_cases.jsonl --gate examples/quality_gate.json
agent-eval-mcp

Case Format

Each JSONL row contains one evaluated agent run:

{
  "case_id": "case-0001",
  "question": "What contribution is Ada Lovelace known for in mathematics?",
  "reference_answer": "Ada Lovelace is known for Analytical Engine notes.",
  "expected_keywords": ["Ada Lovelace", "Analytical Engine"],
  "retrieved_docs": [
    {"doc_id": "wiki-1", "title": "Ada Lovelace", "text": "...", "score": 0.94}
  ],
  "agent_answer": "Ada Lovelace is known for Analytical Engine notes.",
  "latency_ms": 240.5,
  "cost_usd": 0.0031,
  "tags": ["wiki", "rag"]
}

CI Quality Gate

The evaluator exits non-zero when --fail-on-gate is used and thresholds fail:

PYTHONPATH=src python3 -m agent_eval_platform evaluate \
  --cases examples/wiki_eval_cases.jsonl \
  --gate examples/quality_gate.json \
  --fail-on-gate

See .github/workflows/agent-eval.yml for a GitHub Actions example.

MCP-Style Tool Server

Run:

PYTHONPATH=src python3 -m agent_eval_platform.mcp_server

Supported JSON-RPC methods:

  • initialize

  • tools/list

  • tools/call with:

    • run_evaluation_suite

    • compare_regression

    • decide_canary

This is intentionally stdio and dependency-free. It follows the MCP tool shape closely enough for local agent integration demos without requiring the MCP Python SDK.

Canary Workflow

PYTHONPATH=src python3 -m agent_eval_platform canary \
  --result reports/eval_result.json \
  --config examples/canary_config.json \
  --json-out reports/canary_decision.json

The decision is hold, increase_traffic, or promote based on suite quality and minimum case coverage.

Evaluation Results

Evaluation Run

PYTHONPATH=src python3 -m agent_eval_platform evaluate \
  --cases examples/wiki_eval_cases.jsonl \
  --gate examples/quality_gate.json \
  --variant candidate \
  --json-out reports/eval_result.json \
  --markdown-out reports/eval_report.md \
  --traces-out reports/traces.jsonl

Aggregate Metrics

Metric

Value

Evaluation cases

120

Pass rate

82.5%

Average faithfulness

0.825

Average retrieval relevance

0.838

Average hallucination risk

0.153

p50 latency

392.58 ms

p95 latency

663.40 ms

p99 latency

872.87 ms

Average cost

$0.00393

Total cost

$0.47158

Canary Decision

PYTHONPATH=src python3 -m agent_eval_platform canary \
  --result reports/eval_result.json \
  --config examples/canary_config.json \
  --json-out reports/canary_decision.json

Result:

{
  "action": "hold",
  "next_traffic_percent": 10.0,
  "reasons": [
    "pass_rate 0.825 < 0.900"
  ]
}

The canary policy correctly blocked promotion because the candidate run did not meet the configured 90% pass-rate threshold. This demonstrates how the platform can prevent low-quality agent versions from being promoted automatically.

Observability

The evaluation emits OpenTelemetry-style JSONL traces to:

reports/traces.jsonl

Each case generates spans for faithfulness, retrieval relevance, hallucination risk, and final case-level pass/fail status, enabling debugging of failed agent responses.

Available Tools

3 tools
compare_regressionC

Compare baseline and candidate suite JSON files.

Inputs: baseline_path (required), candidate_path (required), out.

decide_canaryC

Apply canary promotion policy to a suite result.

Inputs: config_path (required), result_path (required).

run_evaluation_suiteC

Run RAG/wiki agent evaluation cases and emit JSON, Markdown, and telemetry outputs.

Inputs: cases_path (required), variant, json_out, gate_path, +2 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool has a distinct purpose: running evaluations, comparing results, and applying deployment policy. There is no overlap.

    Naming Consistency5/5

    All tools follow a consistent verb_noun pattern (run_evaluation_suite, compare_regression, decide_canary), making them predictable.

    Tool Count4/5

    With 3 tools, the server covers the core evaluation workflow without being bloated. Slightly minimal but appropriate for a focused server.

    Completeness4/5

    The tools cover the main lifecycle: run suite, compare results, decide promotion. Minor gaps like suite management exist, but the core workflow is complete.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      MCP server that lets coding agents test AI agents. Create YAML test cases, snapshot golden baselines, check for regressions, and generate visual reports all from inside Claude Code or any MCP-compatible tool. Works with LangGraph, CrewAI, OpenAI, Claude, Mistral, and any HTTP API.
      10
      143 npm
      1,798 PyPI
      142
      Apache 2.0
    • A
      license
      A
      quality
      D
      maintenance
      An MCP server that lets any AI agent evaluate RAG outputs -- faithfulness scoring, hallucination detection, and retrieval quality metrics -- with zero API keys, using MCP sampling.
      6
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      A local MCP server that provides tools for AI coding agents to execute, evaluate, and diff regression tests for LLM agents.
      MIT