# AgentOps EvalBench MCP

> MCP-powered LLM evaluation and observability platform for testing RAG and agentic AI systems across groundedness, hallucination risk, retrieval quality, latency, and cost.

**AgentOps EvalBench MCP** is a quality-control platform for LLM applications. It lets developers ingest real TXT, Markdown, and PDF documents, build evaluation test sets, run a RAG pipeline, score generated answers, inspect failed cases, compare prompt/model versions, and export reports. Metrics cover groundedness, hallucination risk, retrieval quality, answer relevance, `retrieval_context_recall`, latency, token usage, and estimated generation cost.

Cost values are **estimated generation-token cost only**, not total provider spend. Offline/extractive generation reports `$0` because no generation tokens are billed; embedding, retrieval, and optional LLM-judge costs are not tracked.

This project focuses on the production layer of AI systems: **evaluation, debugging, observability, and quality gates**.

---

## Live Demo

- **Live Dashboard:** [https://agentops-evalbench-mcp.streamlit.app](https://agentops-evalbench-mcp.streamlit.app)
- **FastAPI Backend:** [https://agentops-evalbench-api.onrender.com](https://agentops-evalbench-api.onrender.com)
- **Interactive Swagger Docs:** [https://agentops-evalbench-api.onrender.com/docs](https://agentops-evalbench-api.onrender.com/docs)
- **Health Check:** [https://agentops-evalbench-api.onrender.com/health](https://agentops-evalbench-api.onrender.com/health)

> **Demo configuration:** The public deployment runs in deterministic offline mode without an OpenAI API key, so it incurs no external LLM inference cost. Persistent application data is stored in Supabase PostgreSQL. The free Render backend may take additional time to wake after inactivity.

## Highlights

- RAG evaluation workflow with project-scoped document loading, retrieval, generation, and scoring
- Real TXT, Markdown, and PDF ingestion, plus bundled sample documents
- Metrics for groundedness, hallucination risk, answer relevance, retrieval quality, `retrieval_context_recall`, latency, token usage, and estimated generation cost
- Failure-safe, project-isolated corpus reindexing and project-owned run artifacts
- Strict prompt versions: omitted versions use `v1`; unsupported explicit versions fail clearly
- Premium Streamlit dashboard for results, failed cases, comparisons, and reports
- FastAPI backend, Typer CLI, and MCP server sharing the same evaluation core
- PostgreSQL persistence with Supabase used only as hosted PostgreSQL through `DATABASE_URL`
- SQLite and deterministic offline fallback mode for local demos and CI without keys
- Offline GitHub Actions regression gate for automated correctness checks

---

## Demo Screenshots

### Dashboard Home

<img width="900" height="890" alt="Dashboard Home" src="https://github.com/user-attachments/assets/aaed705b-a30f-42a1-a90a-72768de21772" />

### Results Dashboard

<img width="1747" height="872" alt="Results Dashboard" src="https://github.com/user-attachments/assets/3698461e-270f-4d54-8394-56a98989654b" />

### Failed Cases

<img width="1675" height="892" alt="Failed Cases" src="https://github.com/user-attachments/assets/65362519-f38e-48d0-ae5e-dd8931687737" />

### Compare Runs

<img width="865" height="875" alt="Compare Runs" src="https://github.com/user-attachments/assets/efbcb741-f2d9-49d3-a805-b5bc2febcda3" />

### Export Report

<img width="606" height="873" alt="Export Report" src="https://github.com/user-attachments/assets/7fe77ae2-aa93-44b5-a2ca-f79d079ccccf" />

---

## How It Works

```text
1. Create a project
2. Load documents or use the included sample documents
3. Create or import evaluation test cases
4. Run the RAG pipeline with OpenAI or deterministic offline extraction
5. Retrieve context and generate answers
6. Score each answer with evaluation metrics
7. Review failed cases and metric breakdowns
8. Compare prompt/model versions
9. Export Markdown or JSON reports
10. Run the workflow through the dashboard, API, CLI, MCP tools, or offline CI gate
```

Each evaluation run stores the question, retrieved context, generated answer, expected answer, metric scores, latency, token usage, estimated generation cost, prompt version, model configuration, pass/fail status, and failure reason.

---

## Results

AgentOps EvalBench MCP includes automated correctness checks and a focused human-labeled evaluator study.

### Software Validation

The repository's current correctness checks are the complete Pytest suite, Ruff, Black, and the deterministic offline CI regression gate described below. Exact test counts are intentionally not hard-coded here so this section does not become stale as coverage grows.

### Evaluator Validation

To test whether the automated evaluator aligns with human judgment, I created a 40-example labeled RAG validation set covering grounded answers, hallucinated answers, partially grounded answers, irrelevant answers, and weak-retrieval cases.

| Metric | Result |
|---|---:|
| Validation set size | 40 examples |
| Pass/fail agreement | 87.5% |
| Groundedness agreement | 90.0% |
| Hallucination precision | 1.000 |
| Hallucination recall | 0.882 |
| Hallucination F1 | 0.938 |

These results show that the evaluator is not only functional as software, but also reasonably aligned with manual review on a focused validation set.

The validation can be reproduced with:

```bash
python -m agentops_evalbench.evaluation.validation
```

## Architecture

```text
                         ┌────────────────────────────────────┐
                         │             Interfaces              │
                         │                                    │
                         │  Streamlit UI   FastAPI   CLI   MCP │
                         └────────┬──────────┬───────┬───────┘
                                  │          │       │
                                  ▼          ▼       ▼
                         ┌────────────────────────────────────┐
                         │        Shared Service Layer          │
                         │ projects / docs / tests / runs /     │
                         │ reports / traces                     │
                         └─────────────────┬──────────────────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              ▼                            ▼                            ▼
    ┌──────────────────┐        ┌────────────────────┐        ┌───────────────────┐
    │   RAG Pipeline   │        │ Evaluation Engine  │        │   Persistence     │
    │ docs → chunks →  │        │ groundedness /     │        │ SQLAlchemy →      │
    │ retrieval → LLM  │───────►│ hallucination /    │───────►│ PostgreSQL        │
    │ answer           │        │ relevance / cost   │        │ SQLite fallback   │
    └──────────────────┘        └────────────────────┘        └───────────────────┘
                                           │
                                           ▼
                                ┌────────────────────┐
                                │ Reports + CI Gate  │
                                │ Markdown / JSON    │
                                └────────────────────┘
```

---

## Tech Stack

| Layer | Technology |
|---|---|
| Dashboard | Streamlit, Plotly, Pandas |
| Backend API | FastAPI, Pydantic, SQLAlchemy, Uvicorn |
| Database | PostgreSQL, Supabase as hosted PostgreSQL, SQLite fallback |
| RAG Pipeline | OpenAI, LangChain, LangGraph, ChromaDB, PyPDF |
| Evaluation | Custom Python evaluators, RAGAS/DeepEval-compatible design |
| CLI | Typer, Rich |
| MCP Server | Python MCP SDK |
| Reports | Markdown, JSON |
| Testing | Pytest, HTTPX |
| Code Quality | Ruff, Black |
| DevOps | Docker, Docker Compose, GitHub Actions |

## Features

- **Document ingestion** — project-scoped TXT, Markdown, and PDF uploads plus bundled samples. Extracted text is persisted so indexes can be rebuilt after restart.
- **Test set manager** — questions, expected answers, ground-truth context.
- **RAG runner** — OpenAI embeddings and generation when configured, with ChromaDB retrieval and deterministic offline fallbacks.
- **Evaluation engine** — groundedness, hallucination risk, answer relevance, retrieval relevance proxy, `retrieval_context_recall`, latency, token usage, and estimated generation cost. Cost covers generation tokens only; embedding, retrieval, and judge costs are not tracked.
- **Corpus isolation** — one deterministic Chroma collection per project; reindexing one project cannot reset another project.
- **Strict prompt versions** — omitted versions use `v1`; unsupported explicit versions fail instead of silently falling back.
- **Thresholds & pass/fail** — configurable via environment variables.
- **Persistence** — projects, documents, test cases, runs, results, and trace logs in PostgreSQL or SQLite.
- **Run isolation** — run details, results, failed cases, traces, and exports validate project ownership.
- **Four interfaces** — FastAPI, Streamlit, CLI, and MCP tools sharing one core.
- **Reports** — Markdown and JSON exports with a pass/fail summary and recommendations.
- **CI quality gate** — GitHub Actions runs an offline deterministic regression gate and fails when quality drops below its checked thresholds.

---

## Project Structure

```text
Agentops-Evalbench-MCP/
├── src/
│   └── agentops_evalbench/
│       ├── api/                 # FastAPI app and routes
│       ├── cli/                 # Typer CLI
│       ├── dashboard/           # Streamlit dashboard
│       ├── evaluation/          # metrics, evaluator, cost tracking
│       ├── mcp_server/          # MCP tools
│       ├── rag/                 # document loading, vector store, RAG pipeline
│       ├── reports/             # Markdown / JSON exporters
│       ├── config.py
│       ├── database.py
│       ├── models.py
│       ├── schemas.py
│       └── services.py
├── data/
│   ├── sample_docs/
│   ├── sample_evals/
│   └── reports/
├── docs/screenshots/
├── tests/
├── .github/workflows/
├── .streamlit/
├── Dockerfile
├── docker-compose.yml
├── pyproject.toml
├── requirements.txt
├── .env.example
└── README.md
```

---

## Setup

Requires **Python 3.10+**.

```bash
git clone https://github.com/AbhinavVarma02/Agentops-Evalbench-MCP.git
cd Agentops-Evalbench-MCP

python -m venv .venv
```

Activate the environment:

```bash
# Windows
.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate
```

Install dependencies:

```bash
# Full install
pip install -r requirements.txt

# Or editable install for development
pip install -e ".[db,dev]"
```

Create a local environment file:

```bash
# Windows
copy .env.example .env

# macOS/Linux
cp .env.example .env
```

Add your values to `.env`:

```env
OPENAI_API_KEY=
DATABASE_URL=
```

`DATABASE_URL` is optional for local testing. If it is missing, the app uses SQLite fallback.

---

## Environment Variables

| Variable | Required | Purpose |
|---|---:|---|
| `OPENAI_API_KEY` | For live LLM runs | OpenAI chat and embeddings |
| `DATABASE_URL` | Recommended | PostgreSQL connection string |
| `DEFAULT_MODEL` | Optional | Defaults to `gpt-4o-mini` |
| `DEFAULT_EMBEDDING_MODEL` | Optional | Defaults to `text-embedding-3-small` |
| `CHROMA_PERSIST_DIR` | Optional | Local vector store path |
| `EVAL_MIN_GROUNDEDNESS` | Optional | Groundedness pass threshold |
| `EVAL_MAX_HALLUCINATION_RISK` | Optional | Hallucination risk threshold |
| `EVAL_MIN_RETRIEVAL_SCORE` | Optional | Retrieval quality threshold |
| `EVAL_MAX_LATENCY_SECONDS` | Optional | Latency threshold |
| `LANGSMITH_API_KEY` | Optional | Tracing support |
| `LANGSMITH_TRACING` | Optional | Enable or disable tracing |

Supabase is used only as hosted PostgreSQL through `DATABASE_URL`. Supabase Auth, Storage, anon keys, and service role keys are not required.

---

## Running the Backend

```bash
python -m uvicorn agentops_evalbench.api.main:app --reload --port 8000
```

Open:

```text
http://127.0.0.1:8000/
http://127.0.0.1:8000/docs
http://127.0.0.1:8000/health
http://127.0.0.1:8000/meta
```

Upload a real project document with multipart form data:

```bash
curl -F "file=@./guide.md" http://localhost:8000/projects/1/documents
```

TXT and Markdown must be UTF-8; PDFs are parsed with PyPDF. The database stores extracted text (not the original binary), which is sufficient for deterministic restart/reindex behavior in the current text-only RAG architecture. Each evaluation reads only its project's persisted corpus. When a test case supplies `ground_truth_context`, the evaluator reports the separate `retrieval_context_recall` metric; the existing retrieval score remains a question/reference-answer relevance proxy.

Uploads are bounded to a 10 MiB body at the application layer: an oversized `Content-Length` is rejected up front, and the endpoint reads at most 10 MiB (+1) before returning `413`. Hard, streaming-level ingress limiting belongs at the reverse proxy / hosting platform in production (this app does not add custom ASGI middleware for it).

**Corpus rebuilds are failure-safe and project-isolated.** Each project's Chroma corpus is content-addressed, so re-indexing after a document change builds and verifies the replacement *before* it becomes active — a failed embed/index leaves the previous corpus usable, and one project's rebuild can never touch another's. A document with no stored persisted content (e.g. a legacy row from before extracted text was persisted) is never silently dropped: indexing/evaluation fails clearly and names the affected documents so they can be re-uploaded.

**Run artifacts are project-scoped.** Run detail, results, failed cases, traces, and exports are served from `GET /projects/{project_id}/eval-runs/{run_id}/…` and validate that the run belongs to the project (a cross-project request returns `404`). The MCP `export_report` / `get_failed_cases` tools likewise require and validate `project_id`.

---

## Running the Dashboard

```bash
streamlit run src/agentops_evalbench/dashboard/streamlit_app.py
```

Open:

```text
http://localhost:8501
```

If the backend is offline, the dashboard shows a friendly offline message with the command to start the API.

---

## Running the CLI

```bash
agentops-eval --help

agentops-eval init
agentops-eval run --project-id 1 --run-name baseline
agentops-eval results --run-id 1
agentops-eval failed --run-id 1
agentops-eval compare --baseline 1 --candidate 2
agentops-eval export --run-id 1 --format markdown
agentops-eval gate --run-id 1 --min-score 0.80
```

---

## Running the MCP Server

```bash
python -m agentops_evalbench.mcp_server.server
```

Available MCP tools:

```text
run_eval
score_answer
compare_runs
export_report
list_eval_runs
get_failed_cases
```

Example MCP server config:

```json
{
  "mcpServers": {
    "agentops-evalbench": {
      "command": "python",
      "args": ["-m", "agentops_evalbench.mcp_server.server"]
    }
  }
}
```

---

## API Endpoints

| Endpoint | Purpose |
|---|---|
| `GET /` | HTML landing page |
| `GET /docs` | Swagger API docs |
| `GET /health` | JSON health check |
| `GET /meta` | API metadata |
| `POST /projects` | Create project |
| `GET /projects` | List projects |
| `POST /projects/{project_id}/documents/load-sample` | Load sample documents |
| `POST /projects/{project_id}/documents` | Upload a TXT, Markdown, or PDF document |
| `GET /projects/{project_id}/documents` | List project documents |
| `POST /projects/{project_id}/documents/reindex` | Rebuild the project corpus |
| `POST /projects/{project_id}/test-cases` | Create test case |
| `GET /projects/{project_id}/test-cases` | List test cases |
| `POST /projects/{project_id}/eval-runs` | Run evaluation |
| `GET /projects/{project_id}/eval-runs/{run_id}` | Get an owned run summary |
| `GET /projects/{project_id}/eval-runs/{run_id}/results` | Get owned run results |
| `GET /projects/{project_id}/eval-runs/{run_id}/failed-cases` | Get owned failed cases |
| `GET /projects/{project_id}/eval-runs/{run_id}/traces` | Get owned run traces |
| `GET /projects/{project_id}/eval-runs/{run_id}/export` | Export an owned run report |
| `POST /eval-runs/compare` | Compare runs |

---

## Running Tests

```bash
pytest
ruff check .
black --check .
```

## Evaluator Validation

A 40-example human-labeled validation set lives at `data/sample_evals/evaluator_validation_set.json`. It covers grounded answers, hallucinated answers, partially grounded answers, irrelevant answers, and weak-retrieval cases.

```bash
python -m agentops_evalbench.evaluation.validation
# or through the Typer CLI
agentops-eval evaluator-validation
```

The study reports 87.5% pass/fail agreement, 90.0% groundedness agreement, 1.000 hallucination precision, 0.882 recall, and 0.938 F1. The runner exports `data/reports/evaluator_validation_results.md` and `data/reports/evaluator_validation_results.json`. For isolated validation without loading local dotenv files, set `AGENTOPS_EVALBENCH_DISABLE_DOTENV=true` before running the command.

## GitHub Actions Quality Gate

[.github/workflows/eval-gate.yml](.github/workflows/eval-gate.yml) installs dependencies, runs Ruff, Black, the complete test suite, and an evaluation gate on sample data. It explicitly disables dotenv loading and clears live credentials:

```yaml
env:
  AGENTOPS_EVALBENCH_DISABLE_DOTENV: "true"
  OPENAI_API_KEY: ""
  DATABASE_URL: sqlite:///./data/ci.db
```

This is an **offline, deterministic regression gate** — a plumbing and quality tripwire for CI, not a measurement of production LLM quality. It checks average groundedness, hallucination risk, retrieval score, answer relevance, and latency, plus a **minimum pass-rate** floor (`--min-pass-rate 0.50`).

The bundled 8-case suite is grounded and answerable, but CI runs fully offline. Its deterministic extractive generator returns correct-but-lexically-divergent answers, so the term-overlap answer-relevance and retrieval proxies produce a reproducible 4/8 pass baseline while every checked average remains above threshold. A real regression trips an average and/or drops the pass rate below `0.50`. This baseline is not a claim that a 50% pass rate reflects production quality.

For evidence of **evaluator quality**, rely on the separate 40-example human-labeled study above. That benchmark measures agreement with human review; the CI gate guards against regressions in the offline pipeline.

---

## What This Project Demonstrates

- LLM evaluation and reliability engineering
- RAG pipeline design
- AI observability and quality gates
- MCP tool integration
- FastAPI backend development
- Streamlit dashboarding
- CLI tooling for developer workflows
- PostgreSQL persistence with SQLAlchemy
- Secure environment variable handling
- Testable and offline-friendly AI system design

---

## Future Improvements

- Add async/batched evaluation for larger test sets
- Add more provider adapters through a pluggable model interface
- Add richer agent trace visualization
- Add user accounts for hosted multi-user usage
- Add a lightweight VS Code extension as a separate phase
- Add deployed demo links after cloud deployment is complete

---

## License

This project is licensed under the MIT License.