Skip to main content
Glama
Sign In
EvalForge Lite

EvalForge Lite

by thejaredchapman
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Hybrid

EvalForge Lite

Compare text LLMs side by side. Write a few test prompts, pick up to four models across OpenRouter, Amazon Bedrock, Google Vertex AI and Microsoft Foundry, and EvalForge Lite sends the same prompts to all of them, scores the answers automatically, and shows a leaderboard with letter grades, response time, speed and estimated cost. Use it as a web app or as an MCP server for Claude and other assistants. You bring your own credentials, or the person hosting it keeps them on the server for you.

Documentation site: https://thejaredchapman.github.io/evalforge-lite/

Why use it

  • Test on your own prompts. Choose models from evidence about your tasks, not a generic benchmark.

  • Up to 4 models per run, across 4 backends. X (OpenRouter) and X@bedrock are separate targets, so you can check one model on two platforms in a single run.

  • Automatic grading. A judge model scores each answer against your rubric, and rule checks (contains, regex, json_valid, max_length, available through the API and MCP) add a pass or fail. You get a 0-100 score and a letter grade.

  • A second opinion on every response. Each answer is also evaluated on six criteria (answered, quality, instruction following, completeness, helpfulness, safety) with strengths and weaknesses written out.

  • Speed and cost beside quality. Latency, tokens per second and estimated cost for every model, and a "What matters most?" selector that moves the "Best for ..." badge without a new run.

  • Policy gate. Upload a company policy and prompts that violate it are blocked before any model is called. If the check itself fails, the prompt is blocked.

  • Reports. Download a PDF or a CSV for any of your last five runs.

  • No accounts, no database. Credentials are used for one request and not stored. Nothing is written to disk.

Related MCP server: BlindWrite MCP

Quick start

1. Run the web app on your computer

Requires Python 3.10 or newer (3.12 recommended; download from https://www.python.org/downloads/) and git.

git clone https://github.com/thejaredchapman/evalforge-lite.git
cd evalforge-lite
python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python app.py

Open http://localhost:8000, paste a key for at least one backend (an OpenRouter API key is the quickest: https://openrouter.ai/workspaces/default/keys), add a test case, pick two to four models, and click Run comparison. Runs are limited to 3 per 8 hours per browser session. Full walkthrough: Getting started.

2. Use it from Claude (MCP server)

With uv installed:

uvx evalforge-lite

Add it to Claude Code in one line:

claude mcp add evalforge-lite -- uvx evalforge-lite

Or install the Claude Code plugin, which bundles the same server:

claude plugin marketplace add thejaredchapman/evalforge-lite
claude plugin install evalforge-lite@evalforge

Then ask your assistant to compare models. It gets 9 tools: list_models, suggest_models, list_availability, set_policy, evaluate_prompt, run_comparison, list_runs, get_report, get_report_csv. Details, Claude Desktop config and credential shapes: MCP server.

3. Host it for other people

Deploy with the included render.yaml (gunicorn, one worker) or any host that can run gunicorn --workers 1 --threads 4 --bind 0.0.0.0:$PORT app:app. By default every visitor supplies their own key. Optionally keep provider keys on the server with environment variables and a shared daily cap (50 per 24 hours by default). Keep it at one worker: all state is in memory per process. Full guide: Hosting and server-side keys.

Good to know

  • The app does not read .env by itself. To use values from it, run set -a; source .env; set +a before python app.py.

  • Each run is limited to 4 models, and each browser session gets 3 runs per 8 hours.

  • All state lives in memory and is cleared when the server restarts. See Privacy and limits.

  • Upgrading from an older version and reading the CSV or API fields? See the notes in Troubleshooting and FAQ.

Backends and credentials

Backend

What you provide

OpenRouter

One API key

Amazon Bedrock

A region, plus a Bedrock API key or AWS access keys (optional session token)

Google Vertex AI

A project id and region, plus an access token or service-account JSON

Microsoft Foundry

A resource name and region, plus an API key or Entra ID access token

A separate judge backend setting chooses where the judge and policy gate run. Bedrock, Vertex and Foundry costs are estimates from catalog prices, not your cloud bill. See Backends and credentials.

Documentation

Page

What is in it

Overview

What it is, who it is for, the three ways to use it

Getting started

Install, run, and your first comparison

Web app guide

Every part of the screen, in order

Comparing models

Reading metrics, grades, evaluation, cost and their limits

Backends and credentials

Keys, regions, X@backend targets

MCP server

Install paths, all 9 tools, example prompts

Hosting and server-side keys

Deploying for others, operator-held keys, daily cap

Troubleshooting and FAQ

Common messages, fixes, and notes on CSV/API field changes

Privacy and limits

What data goes where, what is stored, every limit

Test

pytest tests/ -v

Every model and HTTP call is mocked, so the suite needs no API key and makes no network calls.

Contributing

Contributions are welcome: bug reports, model-catalog updates, new checks, docs, and new backends. See CONTRIBUTING.md for setup, tests and the pull request process. When the app shows an error, the popup's Report an issue on GitHub button opens a pre-filled bug report.

License

MIT

Available Tools

9 tools
evaluate_promptA

Get pre-run feedback on a prompt's clarity/specificity before running a comparison.

An explicit, separately-triggered LLM call (uses your credentials) — not run
automatically as part of run_comparison. Rate-limited independently from
run_comparison's 3-per-8h budget. Pass `creds` as {"openrouter"?: str,
"bedrock"?: {...}, "vertex"?: {...}, "foundry"?: {...}} to use Amazon Bedrock,
Google Vertex AI, or Microsoft Foundry; a bare `api_key` is treated as an
OpenRouter key. `judge_backend` picks which backend runs the evaluation.
If the operator has set server-side keys for a backend (see https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/hosting-and-server-keys.md), those are used for it automatically and creds for it are not needed.

Inputs: prompt (required), creds, api_key, judge_backend.

get_reportA

Get a PDF report (base64-encoded) for a run. Defaults to the most recent run.

priority (balanced|quality|fastest|cheapest), if given, overrides the priority the
run was made with for the report's priority/best-pick line; otherwise the run's own
priority (from run_comparison) is used.

Inputs: run_id, priority.

get_report_csvB

Get a CSV export for a run, one row per (test case x model) cell. Defaults to the most recent run.

Inputs: run_id.

list_availabilityA

Return the current model-availability snapshot: live OpenRouter listing status (refreshed at most every 6 hours) plus curated Bedrock/Vertex/Foundry region coverage.

Inputs: no named parameters.

list_modelsB

List every provider and model in the catalog, plus each provider's frontier (flagship) model.

Each model carries a `reasoning` flag (supports extended reasoning) and a `tags` list of
curated need/industry tag ids; `tags` in the result gives their labels, one-line "why"
explanations and the date they were verified. Tags are a starting point, not benchmarks.

Inputs: no named parameters.

list_runsA

List metadata for the 5 most recent runs, newest first.

Inputs: no named parameters.

run_comparisonA

Run a set of test-case prompts against a set of models, scoring each response.

Each test case may include an optional "rubric" (scored by an LLM judge) and/or
"checks" (rule-based checks). Returns per-model grades, cost/latency stats, and an
overall verdict. Rate-limited to 3 calls per 8 hours.
Models are "<catalog id>" (OpenRouter) or "<catalog id>@bedrock" / "<catalog id>@vertex" /
"<catalog id>@foundry"; pass matching creds ({"openrouter"?, "bedrock"?, "vertex"?, "foundry"?})
or a bare OpenRouter api_key.
judge_backend picks which backend runs the judge and policy gate.
At most 4 models. priority (balanced|quality|fastest|cheapest) ranks the results;
repeats (1-3) re-sends each prompt for timing accuracy. Returns suggestions (same
provider and backend only), advice, ranking, and best_for_priority, plus a per-run
`cost` total and, per cell, an `evaluation` (answered/quality/instruction_following/
completeness/helpfulness/safety scores, strengths, weaknesses, reasoning, overall).
If the operator has set server-side keys for a backend (see https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/hosting-and-server-keys.md), those are used for it automatically and creds for it are not needed.

Inputs: models (required), test_cases (required), creds, api_key, +3 more.

set_policyB

Set the company policy text used to gate prompts before any model is called.

Inputs: policy_text (required).

suggest_modelsC

Suggest sibling models from the same family as the given model id.

Inputs: model_id (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

    TDQS

    A3.7/5.0

    Scored across 9 tools

    Disambiguation4/5

    Most tools have clearly distinct purposes, but the three model-introspection tools (list_models, suggest_models, list_availability) overlap somewhat, and evaluate_prompt vs run_comparison could be momentarily confused since both invoke LLMs. Descriptions do help differentiate them, especially the explicit pre-run vs full-run distinction.

    Naming Consistency5/5

    All tools use snake_case with a consistent verb_noun pattern (list_models, suggest_models, set_policy, evaluate_prompt, run_comparison, list_runs, get_report). Even the format variant get_report_csv follows the same convention cleanly.

    Tool Count5/5

    Nine tools is well-scoped for a model-evaluation server, with each tool earning its place across catalog introspection, policy, evaluation, execution, and reporting. No padding or redundancy in count.

    Completeness4/5

    Coverage spans discovery, policy, prompt feedback, comparison runs, and report exports (PDF/CSV), which is solid for the domain. Minor gaps: no structured JSON run-detail retrieval or run/policy deletion, and set_policy has no corresponding get_policy to inspect current state.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Resources

    Related MCP Connectors

    Related MCP Servers

    • BlindWrite MCP

      ygstudio-game
      A
      license
      A
      quality
      B
      maintenance
      Enables blind, bias-free AI writing evaluations through randomized A/B duels, Bradley-Terry MLE rankings, and preference analytics, while letting Claude delegate heavy drafting to OpenRouter models to save tokens.
      11
      MIT

    • AI Model Experiments

      pipeworx-io
      A
      license
      Not graded
      quality
      C
      maintenance
      Enables running the same prompt(s) across roughly 300 models from Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, Qwen and others, then comparing per-model outputs, latency, token usage, error rates and billed cost side by side. Supports free cost estimates before launching, asynchronous run creation with status polling and cancellation, prepaid x402 credit top-ups, and an optional AI-written comparison summary of the results.
      126 npm
      MIT

    • galton

      Luissalet
      A
      license
      Not graded
      quality
      B
      maintenance
      Enables assistants to queue and monitor local language and vision model evaluations, compare results with confidence intervals, and retrieve routing recommendations for task categories.
      MIT