How do I use mcp-llm-behave?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@mcp-llm-behave Run a behavior test: prompt='Explain AI', expected='define AI', output='AI is...'" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

mcp-llm-behave

by Swanand33

Overview Schema Source Related Servers Discussions Admin

Python

Local

mcp-llm-behave

MCP server exposing llm-behave behavioral regression testing as callable tools inside Claude Desktop, Claude Code, and any MCP-compatible client.

Runs offline — no API calls, no external services. Uses sentence-transformers for embedding-based similarity.

Tools

Tool	What it does
`run_behavior_test`	Assert that a model output matches an expected behavior description
`compare_outputs`	Detect semantic drift between a baseline and a new LLM output
`list_builtin_behaviors`	Browse the built-in behavioral checks shipped with llm-behave

Related MCP server: eval-mcp-server

Quickstart — Claude Desktop

Add to your claude_desktop_config.json (no install needed, uvx handles it):

{
  "mcpServers": {
    "mcp-llm-behave": {
      "command": "uvx",
      "args": ["mcp-llm-behave"]
    }
  }
}

Config file location:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Restart Claude Desktop after editing. The first run downloads the sentence-transformers model (~80 MB) once and caches it.

Quickstart — Claude Code (CLI)

claude mcp add mcp-llm-behave uvx mcp-llm-behave

Install via pip / uv

pip install mcp-llm-behave
# or
uv add mcp-llm-behave

Run the server directly:

mcp-llm-behave

Tool reference

`run_behavior_test`

Check whether a model output semantically satisfies an expected behavior.

Arguments

Name	Type	Description
`prompt`	str	The original prompt sent to the LLM (used for context/logging)
`expected_behavior`	str	Plain-language description of what the output should do
`model_output`	str	The actual text returned by the LLM

Returns

{
  "score": 0.82,
  "passed": true,
  "threshold": 0.45
}

`compare_outputs`

Detect semantic drift between a known-good baseline and a new output. Useful in CI after prompt or model changes.

Arguments

Name	Type	Description
`baseline`	str	The reference / previous LLM output
`candidate`	str	The new LLM output to compare

Returns

{
  "similarity_score": 0.91,
  "drift_detected": false,
  "interpretation": "Outputs are nearly identical — no drift."
}

`list_builtin_behaviors`

Returns the catalog of pre-defined behavioral checks available in llm-behave, with method signatures and descriptions.

Returns — list of objects with name, method, and description keys.

Requirements

Python 3.10+
No API keys needed
~80 MB disk for the sentence-transformers model (downloaded once on first run)

Development

git clone https://github.com/Swanand33/mcp_llm_behave
cd mcp-llm-behave
uv sync
uv run pytest

License

MIT — see LICENSE.

Available Tools

3 tools

compare_outputsA

Compare two LLM outputs for semantic similarity (regression detection).

Useful for catching silent model regressions: run this in CI against a known-good baseline output to detect drift when you change prompts or models.

ParametersJSON Schema

Name	Required	Description	Default
`baseline`	Yes	The reference/previous LLM output.
`candidate`	Yes	The new LLM output to compare against baseline.

Output Schema

ParametersJSON Schema

Name	Required	Description
No output parameters

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must convey behavioral traits. It describes the tool as performing semantic comparison (likely read-only) without side effects, but it does not disclose potential behaviors like auth needs, rate limits, or error handling. The description is adequate but not thorough.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (two sentences) and front-loaded with the core purpose, followed by a usage hint. Every sentence adds value with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity, two parameters, and output schema existence, the description covers the essential context (purpose, usage scenario) without needing to explain return values. It is sufficiently complete for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with both parameters described. The description does not add additional meaning beyond the schema (e.g., expected format or constraints), so it meets the baseline of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: compare two LLM outputs for semantic similarity (regression detection). The verb 'compare' and resource 'outputs' are specific, and it distinguishes from sibling tools like list_builtin_behaviors and run_behavior_test, which serve different functions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear usage context: run in CI against a known-good baseline to detect drift when changing prompts or models. It implies when to use but does not explicitly state when not to use or name alternatives, though the sibling tools hint at different purposes.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_builtin_behaviorsA

Return the catalog of built-in behavioral checks available in llm-behave.

Returns: list of dicts, each with 'name', 'method', and 'description' keys.

ParametersJSON Schema

Name	Required	Description	Default
No parameters

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A3.9/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It discloses the return format (list of dicts with 'name', 'method', 'description') adequately, but does not mention behavioral traits like read-only nature, performance, or error behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, zero waste. The purpose is stated first, followed by concise return structure. Ideal length and structure for a simple tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given zero parameters and an existing output schema, the description provides the return keys. However, it lacks usage context (e.g., when to list vs. run tests) and does not cover edge cases or failure modes. Adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0 parameters with 100% coverage. The description adds no parameter info because none exists, which is appropriate. Baseline of 4 for zero-parameter tools.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Return' and the resource 'catalog of built-in behavioral checks.' It immediately differentiates from sibling tools 'compare_outputs' and 'run_behavior_test' which involve executing or comparing tests rather than listing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use or when-not-to-use guidance is provided. While the zero-parameter signature implies it is a simple retrieval tool, the description does not mention it as a prerequisite for other tools or give any context for decision-making.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_behavior_testA

Run a single behavioral assertion using llm-behave embedding similarity.

Checks whether the model output semantically satisfies the expected behavior. Uses sentence-level max similarity so long outputs are handled correctly.

ParametersJSON Schema

Name	Required	Description
`prompt`	Yes	The original prompt sent to the LLM (used for context/logging).
`expected_behavior`	Yes	A plain-language description of what the output should do.
`model_output`	Yes	The actual text returned by the LLM.

Output Schema

ParametersJSON Schema

Name	Required	Description
No output parameters

TDQS

A3.9/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries the full burden. It discloses that the tool uses 'embedding similarity' and handles long outputs via 'sentence-level max similarity'. This gives meaningful insight into behavior beyond basic expectations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely concise: two short sentences plus a line. Each sentence provides distinct value: purpose, mechanism, and handling of long outputs. No wasted words. Front-loaded with the core action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given three clearly documented parameters in the schema, an output schema (per context), and helpful behavioral details, the description is complete. It covers what the tool does, how it works, and a key edge case (long outputs). No obvious gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and each parameter already has a clear schema description. The tool description restates the purpose of 'prompt' (context/logging) and the nature of 'expected_behavior' (plain-language), but adds no new semantics beyond the schema. Baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear verb and resource: 'Run a single behavioral assertion'. It explains the technique (embedding similarity) and goal (check semantic satisfaction). However, it does not explicitly differentiate from sibling tools 'compare_outputs' and 'list_builtin_behaviors', which would help selection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for checking expected behavior semantically, but lacks explicit guidance on when to use this tool versus alternatives like 'compare_outputs', or when not to use it. No prerequisites or exclusions stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

3 tool updatesv0.1.0Jun 18, 2026
- First observedcompare_outputs
- First observedlist_builtin_behaviors
- First observedrun_behavior_test

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.1/5.0

Scored 2026-06-18 06:31 across 3 tools

Disambiguation5/5

Each tool has a distinct and clearly separated purpose: comparing outputs, listing built-in behaviors, and running a behavior test. No overlap or ambiguity.

Naming Consistency5/5

All tool names follow the same verb_noun pattern in snake_case: compare_outputs, list_builtin_behaviors, run_behavior_test. Perfectly consistent.

Tool Count4/5

Three tools is small but sufficient for the server's focused domain of LLM behavior testing. The set covers the essential operations without being overly minimal.

Completeness4/5

The tools cover listing available behaviors, running a test, and comparing outputs. While additional features like adding custom behaviors could be useful, the core workflow is complete.

Maintenance

ActivityInactive

ResponsivenessNo issues

Resources

Need Help?

Related Servers

Related MCP Connectors

rubrkit B
com.rubrkit
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
AI & Machine Learning Testing & QA Tools
412
Singularity - MCP Discovery & Security Gateway AOAuth
io.github.gaurav-kumar-sinha-060705
Find, vet, and run MCP tools through a secure audited gateway with prompt-injection risk scoring
Security Developer Tools
511
3TG Test Generation A
dev.3tg
3rd Generation Testing (3TG) — generate deterministic test suites from Markdown spec tables via MCP.
Testing & QA Tools Developer Tools
102
geminiupgradeqa-mcp
com.clauxel.geminiupgradeqa
Remote MCP for Gemini upgrade evals, prompt regressions, output diffs, and eval receipts.
Testing & QA Tools AI & Machine Learning
2

Related MCP Servers

Patronus MCP Serverofficial
AI & Machine Learning Testing & QA Tools
patronus-ai
A
license
Not graded
quality
D
maintenance
Enables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.
Updated a year ago (2025-03-26 12:33 UTC)
16
Apache 2.0
eval-mcp-server
AI & Machine Learning RAG Systems
christianmacion
F
license
Not graded
quality
B
maintenance
Exposes a deterministic AI-slop scanner and RAG grounding grader as MCP Tools, Resource, and Prompt, enabling any MCP client to evaluate text quality and context faithfulness.
Updated 2 months ago (2026-07-08 18:01 UTC)
-
mycelium
RAG Systems Search Vector Databases
Transmitt0r
A
license
Not graded
quality
B
maintenance
Provides MCP tools for semantic search over personal knowledge sources using pluggable embeddings and local vector indexing.
Updated a month ago (2026-08-07 01:09 UTC)
1
MIT
agent-eval
Testing & QA Tools AI & Machine Learning
RudrenduPaul
A
license
A
quality
C
maintenance
MCP server exposing statistical regression testing for LLM agents as a "run" tool: p-value, effect size, and confidence interval on whether agent behavior actually changed.
Updated 12 days ago (2026-09-08 22:23 UTC)
1
74 PyPI
Apache 2.0

mcp-llm-behave

Tools

Quickstart — Claude Desktop

Quickstart — Claude Code (CLI)

Install via pip / uv

Tool reference

run_behavior_test

compare_outputs

list_builtin_behaviors

Requirements

Development

License

Available Tools

Output Schema

TDQS

Output Schema

TDQS

Output Schema

TDQS

Tool Schema Changelog

TDQS

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

Patronus MCP Serverofficial

eval-mcp-server

mycelium

agent-eval

`run_behavior_test`

`compare_outputs`

`list_builtin_behaviors`