Skip to main content
Glama

litmiq-ci

test

Open-source CLI, GitHub Action, and MCP connector for Litmiq regression suites. No dependencies. Node 18+.

You export a suite from the Litmiq app (or write a litmiq-suite/1 JSON file), then run the same scoring the app uses: one LLM judge call per output, metrics 0–10, pass = overall ≥ 7. Missing scores are never invented. Provider calls use your OpenAI and/or Anthropic keys; this project does not keep them and does not bill you.

The tool is free. OpenAI or Anthropic bill API usage to your key.

CLI

# from a clone
node litmiq-ci.mjs path/to/suite.json --out litmiq-results.json --baseline litmiq-baseline.json

# check the file only (no keys, no provider calls)
node litmiq-ci.mjs path/to/suite.json --validate

# after install
npx github:litmiq/litmiq-ci -- path/to/suite.json --validate

Flags: --validate, --out file, --baseline file, --update-baseline, --help.

Exit codes: 0 pass, 1 regression or threshold miss, 2 usage or config error, 3 provider errors on every call.

Environment:

  • OPENAI_API_KEY — required when the suite or judge uses an OpenAI model

  • ANTHROPIC_API_KEY — required when the suite or judge uses a Claude model

  • OPENAI_BASE_URL / ANTHROPIC_BASE_URL — optional, for proxies or tests

  • LITMIQ_ALERT_WEBHOOK — optional Slack-compatible incoming webhook, posted to only when a regression is found

  • GITHUB_STEP_SUMMARY — when set (GitHub Actions), the markdown table is appended there

This CLI currently calls the official OpenAI and Anthropic APIs (or a base URL you set). Browser-only providers in the Litmiq app are not run here.

Related MCP server: LLM Evaluation Harness MCP Server

GitHub Action

# .github/workflows/litmiq-evals.yml
name: Litmiq evals
on:
  pull_request:
  push:
    paths: ['litmiq/**']
  schedule:
    - cron: '17 6 * * 1'
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - uses: litmiq/litmiq-ci@v1
        with:
          suite: litmiq/litmiq-suite.json
          baseline: litmiq/litmiq-baseline.json
          openai_api_key: ${{ secrets.OPENAI_API_KEY }}
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

A failed run (regression or provider error) fails the workflow, and GitHub notifies you the usual way. Review the workflow and the suite before you rely on them. You can also download this same CLI from the Litmiq app workspace (Pro can export a scheduled workflow).

MCP connector

A stdio MCP server, no dependencies, exposes three tools: validate_suite, run_suite, and read_results. run_suite uses the same CLI and the keys already in the server environment.

{
  "mcpServers": {
    "litmiq-ci": {
      "command": "node",
      "args": ["/path/to/litmiq-ci/mcp/server.mjs"],
      "env": {
        "OPENAI_API_KEY": "your-key",
        "ANTHROPIC_API_KEY": "your-key"
      }
    }
  }
}

Suite format

schema is litmiq-suite/1. litmiq_fields.system_prompt and cases[].user_input are required. Optional: expected_behavior, models, judge_model, and ci (pass_score, max_avg_drop, max_pass_rate_drop, min_avg). An example is in examples/refund-verification.suite.json. Export one from https://litmiq.com/app.

What this is not

It is not a hosted eval service, not a replacement for the Litmiq app, and not a way to share your keys or suites with Litmiq. Team-wide shared workspaces still need a backend; this repo is the local and CI path.

License

MIT. Copyright (c) 2026 Devansh Kaushik.

Contact: hello@litmiq.com

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Evaluates RAG outputs on faithfulness, answer relevancy, and context precision using an LLM-as-a-Judge backend. Exposes tools for running evaluations, scoring individual samples, and checking thresholds, enabling CI gating and on-demand assessment via MCP.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables MCP clients to evaluate text or local files for LLM-output quality issues, returning 0-100 scores, grades, and precise line/column findings. It also lets clients list all available lint rules and severities.
      MIT