mcp-llm-behave

MCP server exposing llm-behave behavioral regression testing as callable tools inside Claude Desktop, Claude Code, and any MCP-compatible client.

Runs offline — no API calls, no external services. Uses sentence-transformers for embedding-based similarity.

Tools

Related MCP server: AgentOps EvalBench MCP

Quickstart — Claude Desktop

Add to your claude_desktop_config.json (no install needed, uvx handles it):

{
  "mcpServers": {
    "mcp-llm-behave": {
      "command": "uvx",
      "args": ["mcp-llm-behave"]
    }
  }
}

Config file location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

• Windows: %APPDATA%\Claude\claude_desktop_config.json

Restart Claude Desktop after editing. The first run downloads the sentence-transformers model (~80 MB) once and caches it.

Quickstart — Claude Code (CLI)

claude mcp add mcp-llm-behave uvx mcp-llm-behave

Install via pip / uv

pip install mcp-llm-behave
# or
uv add mcp-llm-behave

Run the server directly:

mcp-llm-behave

Tool reference

run_behavior_test

Check whether a model output semantically satisfies an expected behavior.

Arguments

Returns

{
  "score": 0.82,
  "passed": true,
  "threshold": 0.45
}

compare_outputs

Detect semantic drift between a known-good baseline and a new output. Useful in CI after prompt or model changes.

Arguments

Returns

{
  "similarity_score": 0.91,
  "drift_detected": false,
  "interpretation": "Outputs are nearly identical — no drift."
}

list_builtin_behaviors

Returns the catalog of pre-defined behavioral checks available in llm-behave, with method signatures and descriptions.

Returns — list of objects with name, method, and description keys.

Requirements

• Python 3.10+

• No API keys needed

• ~80 MB disk for the sentence-transformers model (downloaded once on first run)

Development

git clone https://github.com/Swanand33/mcp_llm_behave
cd mcp-llm-behave
uv sync
uv run pytest

License

MIT — see LICENSE.