litmiq-ci
OfficialRuns LLM-judge scoring calls against the official OpenAI API using your own OPENAI_API_KEY (optionally via OPENAI_BASE_URL), so suite outputs can be evaluated against a baseline without the server ever storing your key.
Posts an alert to a Slack-compatible incoming webhook (LITMIQ_ALERT_WEBHOOK) only when a regression is detected in a suite run.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
litmiq-ci
Open-source CLI, GitHub Action, and MCP connector for Litmiq regression suites. No dependencies. Node 18+.
You export a suite from the Litmiq app (or write a litmiq-suite/1 JSON file), then run the same scoring the app uses: one LLM judge call per output, metrics 0–10, pass = overall ≥ 7. Missing scores are never invented. Provider calls use your OpenAI and/or Anthropic keys; this project does not keep them and does not bill you.
The tool is free. OpenAI or Anthropic bill API usage to your key.
CLI
# from a clone
node litmiq-ci.mjs path/to/suite.json --out litmiq-results.json --baseline litmiq-baseline.json
# check the file only (no keys, no provider calls)
node litmiq-ci.mjs path/to/suite.json --validate
# after install
npx github:litmiq/litmiq-ci -- path/to/suite.json --validateFlags: --validate, --out file, --baseline file, --update-baseline, --help.
Exit codes: 0 pass, 1 regression or threshold miss, 2 usage or config error, 3 provider errors on every call.
Environment:
OPENAI_API_KEY— required when the suite or judge uses an OpenAI modelANTHROPIC_API_KEY— required when the suite or judge uses a Claude modelOPENAI_BASE_URL/ANTHROPIC_BASE_URL— optional, for proxies or testsLITMIQ_ALERT_WEBHOOK— optional Slack-compatible incoming webhook, posted to only when a regression is foundGITHUB_STEP_SUMMARY— when set (GitHub Actions), the markdown table is appended there
This CLI currently calls the official OpenAI and Anthropic APIs (or a base URL you set). Browser-only providers in the Litmiq app are not run here.
Related MCP server: LLM Evaluation Harness MCP Server
GitHub Action
# .github/workflows/litmiq-evals.yml
name: Litmiq evals
on:
pull_request:
push:
paths: ['litmiq/**']
schedule:
- cron: '17 6 * * 1'
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- uses: litmiq/litmiq-ci@v1
with:
suite: litmiq/litmiq-suite.json
baseline: litmiq/litmiq-baseline.json
openai_api_key: ${{ secrets.OPENAI_API_KEY }}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}A failed run (regression or provider error) fails the workflow, and GitHub notifies you the usual way. Review the workflow and the suite before you rely on them. You can also download this same CLI from the Litmiq app workspace (Pro can export a scheduled workflow).
MCP connector
A stdio MCP server, no dependencies, exposes three tools: validate_suite, run_suite, and read_results. run_suite uses the same CLI and the keys already in the server environment.
{
"mcpServers": {
"litmiq-ci": {
"command": "node",
"args": ["/path/to/litmiq-ci/mcp/server.mjs"],
"env": {
"OPENAI_API_KEY": "your-key",
"ANTHROPIC_API_KEY": "your-key"
}
}
}
}Suite format
schema is litmiq-suite/1. litmiq_fields.system_prompt and cases[].user_input are required. Optional: expected_behavior, models, judge_model, and ci (pass_score, max_avg_drop, max_pass_rate_drop, min_avg). An example is in examples/refund-verification.suite.json. Export one from https://litmiq.com/app.
What this is not
It is not a hosted eval service, not a replacement for the Litmiq app, and not a way to share your keys or suites with Litmiq. Team-wide shared workspaces still need a backend; this repo is the local and CI path.
License
MIT. Copyright (c) 2026 Devansh Kaushik.
Contact: hello@litmiq.com
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Remote MCP for Gemini upgrade evals, prompt regressions, output diffs, and eval receipts.
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Related MCP Servers
AlicenseNot gradedqualityDmaintenanceEnables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.16Apache 2.0- AlicenseNot gradedqualityDmaintenanceEvaluates RAG outputs on faithfulness, answer relevancy, and context precision using an LLM-as-a-Judge backend. Exposes tools for running evaluations, scoring individual samples, and checking thresholds, enabling CI gating and on-demand assessment via MCP.MIT
- AlicenseAqualityBmaintenanceEnables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.8MIT
- AlicenseNot gradedqualityBmaintenanceEnables MCP clients to evaluate text or local files for LLM-output quality issues, returning 0-100 scores, grades, and precise line/column findings. It also lets clients list all available lint rules and severities.MIT