Skip to main content
Glama
AbhinavVarma02

AgentOps EvalBench MCP

by AbhinavVarma02
OverviewSchemaRelated ServersScoreDiscussions
Python
Hybrid

AgentOps EvalBench MCP

MCP-powered LLM evaluation and observability platform for testing RAG and agentic AI systems across groundedness, hallucination risk, retrieval quality, latency, and cost.

AgentOps EvalBench MCP is a quality-control platform for LLM applications. It helps developers test whether a RAG or agentic AI system is reliable by running evaluation test cases, scoring generated answers, highlighting failed cases, comparing prompt/model versions, and exporting reports.

This project focuses on the production layer of AI systems: evaluation, debugging, observability, and quality gates.

Highlights

  • RAG evaluation workflow with document loading, retrieval, generation, and scoring

  • Metrics for groundedness, hallucination risk, relevance, retrieval quality, latency, token usage, and estimated cost

  • Premium Streamlit dashboard for results, failed cases, comparisons, and reports

  • FastAPI backend for projects, test cases, evaluation runs, results, and exports

  • Typer CLI for local developer workflows and CI usage

  • MCP server exposing evaluation tools through a standard tool interface

  • PostgreSQL persistence with Supabase used only as hosted PostgreSQL through DATABASE_URL

  • SQLite and offline fallback mode for local demos without keys

  • GitHub Actions quality gate for automated checks

Related MCP server: mcp-llm-eval

Demo Screenshots

Dashboard Home

Results Dashboard

Failed Cases

Compare Runs

Export Report

How It Works

1. Create a project
2. Load documents or use the included sample documents
3. Create or import evaluation test cases
4. Run the RAG pipeline
5. Retrieve context and generate answers
6. Score each answer with evaluation metrics
7. Review failed cases and metric breakdowns
8. Compare prompt/model versions
9. Export Markdown or JSON reports
10. Run the workflow through the dashboard, API, CLI, or MCP tools

Each evaluation run stores the question, retrieved context, generated answer, expected answer, metric scores, latency, token usage, estimated cost, prompt version, model configuration, pass/fail status, and failure reason.

Results

AgentOps EvalBench MCP was validated with both software tests and a small human-labeled evaluator study.

Software Validation

Check

Result

Automated tests

31 passed

CLI smoke test

Passed

FastAPI smoke test

Passed

Streamlit dashboard smoke test

Passed

Sample evaluation suite

8 cases

Evaluator Validation

To test whether the automated evaluator aligns with human judgment, I created a 40-example labeled RAG validation set covering grounded answers, hallucinated answers, partially grounded answers, irrelevant answers, and weak-retrieval cases.

Metric

Result

Validation set size

40 examples

Pass/fail agreement

87.5%

Groundedness agreement

90.0%

Hallucination precision

1.000

Hallucination recall

0.882

Hallucination F1

0.938

These results show that the evaluator is not only functional as software, but also reasonably aligned with manual review on a focused validation set.

The validation can be reproduced with:

python -m agentops_evalbench.evaluation.validation

## Architecture

```text
                         ┌────────────────────────────────────┐
                         │             Interfaces              │
                         │                                    │
                         │  Streamlit UI   FastAPI   CLI   MCP │
                         └────────┬──────────┬───────┬───────┘
                                  │          │       │
                                  ▼          ▼       ▼
                         ┌────────────────────────────────────┐
                         │        Shared Service Layer          │
                         │ projects / docs / tests / runs /     │
                         │ reports / traces                     │
                         └─────────────────┬──────────────────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              ▼                            ▼                            ▼
    ┌──────────────────┐        ┌────────────────────┐        ┌───────────────────┐
    │   RAG Pipeline   │        │ Evaluation Engine  │        │   Persistence     │
    │ docs → chunks →  │        │ groundedness /     │        │ SQLAlchemy →      │
    │ retrieval → LLM  │───────►│ hallucination /    │───────►│ PostgreSQL        │
    │ answer           │        │ relevance / cost   │        │ SQLite fallback   │
    └──────────────────┘        └────────────────────┘        └───────────────────┘
                                           │
                                           ▼
                                ┌────────────────────┐
                                │ Reports + CI Gate  │
                                │ Markdown / JSON    │
                                └────────────────────┘

Tech Stack

Layer

Technology

Dashboard

Streamlit, Plotly, Pandas

Backend API

FastAPI, Pydantic, SQLAlchemy, Uvicorn

Database

PostgreSQL, Supabase as hosted PostgreSQL, SQLite fallback

RAG Pipeline

OpenAI, LangChain, LangGraph, ChromaDB, PyPDF

Evaluation

Custom Python evaluators, RAGAS/DeepEval-compatible design

CLI

Typer, Rich

MCP Server

Python MCP SDK

Reports

Markdown, JSON

Testing

Pytest, HTTPX

Code Quality

Ruff, Black

DevOps

Docker, Docker Compose, GitHub Actions

Project Structure

Agentops-Evalbench-MCP/
├── src/
│   └── agentops_evalbench/
│       ├── api/                 # FastAPI app and routes
│       ├── cli/                 # Typer CLI
│       ├── dashboard/           # Streamlit dashboard
│       ├── evaluation/          # metrics, evaluator, cost tracking
│       ├── mcp_server/          # MCP tools
│       ├── rag/                 # document loading, vector store, RAG pipeline
│       ├── reports/             # Markdown / JSON exporters
│       ├── config.py
│       ├── database.py
│       ├── models.py
│       ├── schemas.py
│       └── services.py
├── data/
│   ├── sample_docs/
│   ├── sample_evals/
│   └── reports/
├── docs/screenshots/
├── tests/
├── .github/workflows/
├── .streamlit/
├── Dockerfile
├── docker-compose.yml
├── pyproject.toml
├── requirements.txt
├── .env.example
└── README.md

Setup

Requires Python 3.10+.

git clone https://github.com/AbhinavVarma02/Agentops-Evalbench-MCP.git
cd Agentops-Evalbench-MCP

python -m venv .venv

Activate the environment:

# Windows
.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate

Install dependencies:

# Full install
pip install -r requirements.txt

# Or editable install for development
pip install -e ".[db,dev]"

Create a local environment file:

# Windows
copy .env.example .env

# macOS/Linux
cp .env.example .env

Add your values to .env:

OPENAI_API_KEY=
DATABASE_URL=

DATABASE_URL is optional for local testing. If it is missing, the app uses SQLite fallback.

Environment Variables

Variable

Required

Purpose

OPENAI_API_KEY

For live LLM runs

OpenAI chat and embeddings

DATABASE_URL

Recommended

PostgreSQL connection string

DEFAULT_MODEL

Optional

Defaults to gpt-4o-mini

DEFAULT_EMBEDDING_MODEL

Optional

Defaults to text-embedding-3-small

CHROMA_PERSIST_DIR

Optional

Local vector store path

EVAL_MIN_GROUNDEDNESS

Optional

Groundedness pass threshold

EVAL_MAX_HALLUCINATION_RISK

Optional

Hallucination risk threshold

EVAL_MIN_RETRIEVAL_SCORE

Optional

Retrieval quality threshold

EVAL_MAX_LATENCY_SECONDS

Optional

Latency threshold

LANGSMITH_API_KEY

Optional

Tracing support

LANGSMITH_TRACING

Optional

Enable or disable tracing

Supabase is used only as hosted PostgreSQL through DATABASE_URL. Supabase Auth, Storage, anon keys, and service role keys are not required.

Running the Backend

python -m uvicorn agentops_evalbench.api.main:app --reload --port 8000

Open:

http://127.0.0.1:8000/
http://127.0.0.1:8000/docs
http://127.0.0.1:8000/health
http://127.0.0.1:8000/meta

Running the Dashboard

streamlit run src/agentops_evalbench/dashboard/streamlit_app.py

Open:

http://localhost:8501

If the backend is offline, the dashboard shows a friendly offline message with the command to start the API.

Running the CLI

agentops-eval --help

agentops-eval init
agentops-eval run --project-id 1 --run-name baseline
agentops-eval results --run-id 1
agentops-eval failed --run-id 1
agentops-eval compare --baseline 1 --candidate 2
agentops-eval export --run-id 1 --format markdown
agentops-eval gate --run-id 1 --min-score 0.80

Running the MCP Server

python -m agentops_evalbench.mcp_server.server

Available MCP tools:

run_eval
score_answer
compare_runs
export_report
list_eval_runs
get_failed_cases

Example MCP server config:

{
  "mcpServers": {
    "agentops-evalbench": {
      "command": "python",
      "args": ["-m", "agentops_evalbench.mcp_server.server"]
    }
  }
}

API Endpoints

Endpoint

Purpose

GET /

HTML landing page

GET /docs

Swagger API docs

GET /health

JSON health check

GET /meta

API metadata

POST /projects

Create project

GET /projects

List projects

POST /projects/{project_id}/documents/load-sample

Load sample documents

POST /projects/{project_id}/test-cases

Create test case

GET /projects/{project_id}/test-cases

List test cases

POST /projects/{project_id}/eval-runs

Run evaluation

GET /eval-runs/{run_id}

Get run summary

GET /eval-runs/{run_id}/results

Get detailed results

GET /eval-runs/{run_id}/failed-cases

Get failed cases

GET /eval-runs/{run_id}/export

Export report

POST /eval-runs/compare

Compare runs

Running Tests

pytest
ruff check .
black --check .

Current validation:

31 passed
ruff clean
black clean

GitHub Actions Quality Gate

The repository includes a lightweight CI workflow that installs dependencies, runs tests, and runs a sample evaluation gate.

.github/workflows/eval-gate.yml

What This Project Demonstrates

  • LLM evaluation and reliability engineering

  • RAG pipeline design

  • AI observability and quality gates

  • MCP tool integration

  • FastAPI backend development

  • Streamlit dashboarding

  • CLI tooling for developer workflows

  • PostgreSQL persistence with SQLAlchemy

  • Secure environment variable handling

  • Testable and offline-friendly AI system design

Future Improvements

  • Add async/batched evaluation for larger test sets

  • Add more provider adapters through a pluggable model interface

  • Add richer agent trace visualization

  • Add user accounts for hosted multi-user usage

  • Add a lightweight VS Code extension as a separate phase

  • Add deployed demo links after cloud deployment is complete

License

This project is licensed under the MIT License.

This server cannot be installed

A
license - permissive license
-
quality - not tested
C
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/AbhinavVarma02/Agentops-Evalbench-MCP'

If you have feedback or need assistance with the MCP directory API, please join our Discord server