AgentOps EvalBench MCP

MCP-powered LLM evaluation and observability platform for testing RAG and agentic AI systems across groundedness, hallucination risk, retrieval quality, latency, and cost.

AgentOps EvalBench MCP is a quality-control platform for LLM applications. It lets developers ingest real TXT, Markdown, and PDF documents, build evaluation test sets, run a RAG pipeline, score generated answers, inspect failed cases, compare prompt/model versions, and export reports. Metrics cover groundedness, hallucination risk, retrieval quality, answer relevance, retrieval_context_recall, latency, token usage, and estimated generation cost.

Cost values are estimated generation-token cost only, not total provider spend. Offline/extractive generation reports $0 because no generation tokens are billed; embedding, retrieval, and optional LLM-judge costs are not tracked.

This project focuses on the production layer of AI systems: evaluation, debugging, observability, and quality gates.

Highlights

• RAG evaluation workflow with project-scoped document loading, retrieval, generation, and scoring

• Real TXT, Markdown, and PDF ingestion, plus bundled sample documents

• Metrics for groundedness, hallucination risk, answer relevance, retrieval quality, retrieval_context_recall, latency, token usage, and estimated generation cost

• Failure-safe, project-isolated corpus reindexing and project-owned run artifacts

• Strict prompt versions: omitted versions use v1; unsupported explicit versions fail clearly

• Premium Streamlit dashboard for results, failed cases, comparisons, and reports

• FastAPI backend, Typer CLI, and MCP server sharing the same evaluation core

• PostgreSQL persistence with Supabase used only as hosted PostgreSQL through DATABASE_URL

• SQLite and deterministic offline fallback mode for local demos and CI without keys

• Offline GitHub Actions regression gate for automated correctness checks

Related MCP server: Patronus MCP Server

Demo Screenshots

Dashboard Home

Results Dashboard

Failed Cases

Compare Runs

Export Report

How It Works

1. Create a project
2. Load documents or use the included sample documents
3. Create or import evaluation test cases
4. Run the RAG pipeline with OpenAI or deterministic offline extraction
5. Retrieve context and generate answers
6. Score each answer with evaluation metrics
7. Review failed cases and metric breakdowns
8. Compare prompt/model versions
9. Export Markdown or JSON reports
10. Run the workflow through the dashboard, API, CLI, MCP tools, or offline CI gate

Each evaluation run stores the question, retrieved context, generated answer, expected answer, metric scores, latency, token usage, estimated generation cost, prompt version, model configuration, pass/fail status, and failure reason.

Results

AgentOps EvalBench MCP includes automated correctness checks and a focused human-labeled evaluator study.

Software Validation

The repository's current correctness checks are the complete Pytest suite, Ruff, Black, and the deterministic offline CI regression gate described below. Exact test counts are intentionally not hard-coded here so this section does not become stale as coverage grows.

Evaluator Validation

To test whether the automated evaluator aligns with human judgment, I created a 40-example labeled RAG validation set covering grounded answers, hallucinated answers, partially grounded answers, irrelevant answers, and weak-retrieval cases.

These results show that the evaluator is not only functional as software, but also reasonably aligned with manual review on a focused validation set.

The validation can be reproduced with:

python -m agentops_evalbench.evaluation.validation

Architecture

                         ┌────────────────────────────────────┐
                         │             Interfaces              │
                         │                                    │
                         │  Streamlit UI   FastAPI   CLI   MCP │
                         └────────┬──────────┬───────┬───────┘
                                  │          │       │
                                  ▼          ▼       ▼
                         ┌────────────────────────────────────┐
                         │        Shared Service Layer          │
                         │ projects / docs / tests / runs /     │
                         │ reports / traces                     │
                         └─────────────────┬──────────────────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              ▼                            ▼                            ▼
    ┌──────────────────┐        ┌────────────────────┐        ┌───────────────────┐
    │   RAG Pipeline   │        │ Evaluation Engine  │        │   Persistence     │
    │ docs → chunks →  │        │ groundedness /     │        │ SQLAlchemy →      │
    │ retrieval → LLM  │───────►│ hallucination /    │───────►│ PostgreSQL        │
    │ answer           │        │ relevance / cost   │        │ SQLite fallback   │
    └──────────────────┘        └────────────────────┘        └───────────────────┘
                                           │
                                           ▼
                                ┌────────────────────┐
                                │ Reports + CI Gate  │
                                │ Markdown / JSON    │
                                └────────────────────┘

Tech Stack

Features

• Document ingestion — project-scoped TXT, Markdown, and PDF uploads plus bundled samples. Extracted text is persisted so indexes can be rebuilt after restart.

• Test set manager — questions, expected answers, ground-truth context.

• RAG runner — OpenAI embeddings and generation when configured, with ChromaDB retrieval and deterministic offline fallbacks.

• Evaluation engine — groundedness, hallucination risk, answer relevance, retrieval relevance proxy, retrieval_context_recall, latency, token usage, and estimated generation cost. Cost covers generation tokens only; embedding, retrieval, and judge costs are not tracked.

• Corpus isolation — one deterministic Chroma collection per project; reindexing one project cannot reset another project.

• Strict prompt versions — omitted versions use v1; unsupported explicit versions fail instead of silently falling back.

• Thresholds & pass/fail — configurable via environment variables.

• Persistence — projects, documents, test cases, runs, results, and trace logs in PostgreSQL or SQLite.

• Run isolation — run details, results, failed cases, traces, and exports validate project ownership.

• Four interfaces — FastAPI, Streamlit, CLI, and MCP tools sharing one core.

• Reports — Markdown and JSON exports with a pass/fail summary and recommendations.

• CI quality gate — GitHub Actions runs an offline deterministic regression gate and fails when quality drops below its checked thresholds.

Project Structure

Agentops-Evalbench-MCP/
├── src/
│   └── agentops_evalbench/
│       ├── api/                 # FastAPI app and routes
│       ├── cli/                 # Typer CLI
│       ├── dashboard/           # Streamlit dashboard
│       ├── evaluation/          # metrics, evaluator, cost tracking
│       ├── mcp_server/          # MCP tools
│       ├── rag/                 # document loading, vector store, RAG pipeline
│       ├── reports/             # Markdown / JSON exporters
│       ├── config.py
│       ├── database.py
│       ├── models.py
│       ├── schemas.py
│       └── services.py
├── data/
│   ├── sample_docs/
│   ├── sample_evals/
│   └── reports/
├── docs/screenshots/
├── tests/
├── .github/workflows/
├── .streamlit/
├── Dockerfile
├── docker-compose.yml
├── pyproject.toml
├── requirements.txt
├── .env.example
└── README.md

Setup

Requires Python 3.10+.

git clone https://github.com/AbhinavVarma02/Agentops-Evalbench-MCP.git
cd Agentops-Evalbench-MCP

python -m venv .venv

Activate the environment:

# Windows
.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate

Install dependencies:

# Full install
pip install -r requirements.txt

# Or editable install for development
pip install -e ".[db,dev]"

Create a local environment file:

# Windows
copy .env.example .env

# macOS/Linux
cp .env.example .env

Add your values to .env:

OPENAI_API_KEY=
DATABASE_URL=

DATABASE_URL is optional for local testing. If it is missing, the app uses SQLite fallback.

Environment Variables

Supabase is used only as hosted PostgreSQL through DATABASE_URL. Supabase Auth, Storage, anon keys, and service role keys are not required.

Running the Backend

python -m uvicorn agentops_evalbench.api.main:app --reload --port 8000

Open:

http://127.0.0.1:8000/
http://127.0.0.1:8000/docs
http://127.0.0.1:8000/health
http://127.0.0.1:8000/meta

Upload a real project document with multipart form data:

curl -F "file=@./guide.md" http://localhost:8000/projects/1/documents

TXT and Markdown must be UTF-8; PDFs are parsed with PyPDF. The database stores extracted text (not the original binary), which is sufficient for deterministic restart/reindex behavior in the current text-only RAG architecture. Each evaluation reads only its project's persisted corpus. When a test case supplies ground_truth_context, the evaluator reports the separate retrieval_context_recall metric; the existing retrieval score remains a question/reference-answer relevance proxy.

Uploads are bounded to a 10 MiB body at the application layer: an oversized Content-Length is rejected up front, and the endpoint reads at most 10 MiB (+1) before returning 413. Hard, streaming-level ingress limiting belongs at the reverse proxy / hosting platform in production (this app does not add custom ASGI middleware for it).

Corpus rebuilds are failure-safe and project-isolated. Each project's Chroma corpus is content-addressed, so re-indexing after a document change builds and verifies the replacement before it becomes active — a failed embed/index leaves the previous corpus usable, and one project's rebuild can never touch another's. A document with no stored persisted content (e.g. a legacy row from before extracted text was persisted) is never silently dropped: indexing/evaluation fails clearly and names the affected documents so they can be re-uploaded.

Run artifacts are project-scoped. Run detail, results, failed cases, traces, and exports are served from GET /projects/{project_id}/eval-runs/{run_id}/… and validate that the run belongs to the project (a cross-project request returns 404). The MCP export_report / get_failed_cases tools likewise require and validate project_id.

Running the Dashboard

streamlit run src/agentops_evalbench/dashboard/streamlit_app.py

Open:

http://localhost:8501

If the backend is offline, the dashboard shows a friendly offline message with the command to start the API.

Running the CLI

agentops-eval --help

agentops-eval init
agentops-eval run --project-id 1 --run-name baseline
agentops-eval results --run-id 1
agentops-eval failed --run-id 1
agentops-eval compare --baseline 1 --candidate 2
agentops-eval export --run-id 1 --format markdown
agentops-eval gate --run-id 1 --min-score 0.80

Running the MCP Server

python -m agentops_evalbench.mcp_server.server

Available MCP tools:

run_eval
score_answer
compare_runs
export_report
list_eval_runs
get_failed_cases

Example MCP server config:

{
  "mcpServers": {
    "agentops-evalbench": {
      "command": "python",
      "args": ["-m", "agentops_evalbench.mcp_server.server"]
    }
  }
}

API Endpoints

Running Tests

pytest
ruff check .
black --check .

Evaluator Validation

A 40-example human-labeled validation set lives at data/sample_evals/evaluator_validation_set.json. It covers grounded answers, hallucinated answers, partially grounded answers, irrelevant answers, and weak-retrieval cases.

python -m agentops_evalbench.evaluation.validation
# or through the Typer CLI
agentops-eval evaluator-validation

The study reports 87.5% pass/fail agreement, 90.0% groundedness agreement, 1.000 hallucination precision, 0.882 recall, and 0.938 F1. The runner exports data/reports/evaluator_validation_results.md and data/reports/evaluator_validation_results.json. For isolated validation without loading local dotenv files, set AGENTOPS_EVALBENCH_DISABLE_DOTENV=true before running the command.

GitHub Actions Quality Gate

.github/workflows/eval-gate.yml installs dependencies, runs Ruff, Black, the complete test suite, and an evaluation gate on sample data. It explicitly disables dotenv loading and clears live credentials:

env:
  AGENTOPS_EVALBENCH_DISABLE_DOTENV: "true"
  OPENAI_API_KEY: ""
  DATABASE_URL: sqlite:///./data/ci.db

This is an offline, deterministic regression gate — a plumbing and quality tripwire for CI, not a measurement of production LLM quality. It checks average groundedness, hallucination risk, retrieval score, answer relevance, and latency, plus a minimum pass-rate floor (--min-pass-rate 0.50).

The bundled 8-case suite is grounded and answerable, but CI runs fully offline. Its deterministic extractive generator returns correct-but-lexically-divergent answers, so the term-overlap answer-relevance and retrieval proxies produce a reproducible 4/8 pass baseline while every checked average remains above threshold. A real regression trips an average and/or drops the pass rate below 0.50. This baseline is not a claim that a 50% pass rate reflects production quality.

For evidence of evaluator quality, rely on the separate 40-example human-labeled study above. That benchmark measures agreement with human review; the CI gate guards against regressions in the offline pipeline.

What This Project Demonstrates

• LLM evaluation and reliability engineering

• RAG pipeline design

• AI observability and quality gates

• MCP tool integration

• FastAPI backend development

• Streamlit dashboarding

• CLI tooling for developer workflows

• PostgreSQL persistence with SQLAlchemy

• Secure environment variable handling

• Testable and offline-friendly AI system design

Future Improvements

• Add async/batched evaluation for larger test sets

• Add more provider adapters through a pluggable model interface

• Add richer agent trace visualization

• Add user accounts for hosted multi-user usage

• Add a lightweight VS Code extension as a separate phase

• Add deployed demo links after cloud deployment is complete

License

This project is licensed under the MIT License.