What can you do with this server?

This server exposes a fake order/refund API for testing tool-using AI agents. * get_order: fetch an order by its id. * list_orders: list orders, optionally filtered by customer_id. * create_return: start a return for a given order_id (custom business rule). * issue_refund: create a refund record, optionally with amount, order_id, and created_at. You can use these tools to simulate order lookup, return creation, and refund issuance—including fault-injected failures—and then assert on the resulting state from tests.

How do I use faultbench?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@faultbench run the refund agent test against the shop world with timeout faults" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

faultbench

by mit-hun-k

Overview Schema Source Related Servers Discussions Admin

Python

Hybrid

faultbench

M8ven Score

Fake, stateful worlds with fault injection for testing tool-using AI agents.

faultbench injecting a refund timeout and catching the resulting double refund

Declare your services in a YAML file. faultbench serves them as MCP tools your agent can call, makes them slow or broken on purpose, records every call, and lets you assert on the world's end state from pytest, twenty runs at a time.

Status: pre-alpha, but it works end to end. Write a world, serve it over MCP (stdio or HTTP), inject faults, and assert on state from pytest across N seeded runs. Guides: write a world · test an agent · examples: examples/shop (refund agent), examples/bank. Validated against Pydantic AI and the OpenAI Agents SDK (any MCP framework works).

The problem

Your support agent handles "return my order and refund me." It works when you test by hand. In production the refund API times out once, the agent retries, and a customer is refunded twice. You can't make the real payments API time out on command, so you never tested it.

Related MCP server: datalox-gated-runtime

What it looks like

# world.yaml
services:
  orders:
    records: { order: { id: str, status: enum[placed, delivered, returned], total: float } }
    operations: { get_order: { kind: get, record: order } }
  payments:
    records: { refund: { id: str, order_id: str, amount: float } }
    operations: { issue_refund: { kind: create, record: refund } }
faults:
  payments.issue_refund: { errors: { timeout: 0.10 } }

from faultbench.integrations.pydantic_ai import run_agent  # or wire any MCP framework


@pytest.mark.world("world.yaml")
@pytest.mark.faults("world.yaml")  # inject the faults: block above
@pytest.mark.runs(20)
@pytest.mark.min_pass_rate(0.95)
async def test_refund_issued_exactly_once(world, mcp_server, trace):
    order = world.orders.pick(status="delivered")
    await run_agent(
        "openai:gpt-5-mini",
        f"Return order {order.id} and refund me",
        mcp=mcp_server,
        system_prompt="You are a refund agent.",
    )
    assert len(world.refunds.where(order_id=order.id)) == 1  # a timeout+retry breaks this

faultbench: pass rate over runs
test_refund_issued_exactly_once: 17/20 passed (85%)  min_pass_rate=95% -> FAIL
    run4:  get_order → create_return → issue_refund!timeout → issue_refund
    run11: get_order → create_return → issue_refund!timeout → issue_refund
    run18: get_order → create_return → issue_refund!timeout → issue_refund

The timeout fired after the refund was written, the agent retried, and the customer was refunded twice — the production bug you couldn't trigger on the real payments API, now a red test with the trace that explains it.

What it can and can't model

faultbench models services as flat records (fields: str/int/float/bool/datetime/enum/ref) with built-in CRUD plus custom Python operations for anything else.

Fits well: entities with enums and ref relationships; CRUD and list-by-field; business rules, state machines, and multi-record writes as custom handlers; array inputs via a custom op that flattens into a related record type; money as integer minor-units. (See examples/stripe — a Stripe-style payments API with partial-refund rules.)
Caveat: records are flat — there are no nested objects or array fields. Model a one-to-many as a related record type + a ref (invoice ← line items); a GET returns the parent without children inline, so if your agent's correctness depends on a nested response shape, the fake's shape differs.
Not in 0.1: pagination/cursors, non-equality filters, auth, webhooks, per-request idempotency (that last is a bug faultbench helps you catch, not prevent). Generated seed values are type-correct but not domain-aware — set realistic values in a handler or your test.

Security

A world file can name Python to import and run (custom handlers, handler: module.func), so loading or serving one executes that code. Only use world files you trust, like any script.

Not in scope

Simulated users, LLM judges, dashboards. Use LangWatch Scenario / DeepEval for users and your own judge for scoring; faultbench is the environment.

Quickstart

uv sync --all-extras
uv run pytest                                             # the harness test suite (no API key)
uv run faultbench serve examples/shop/worlds/shop.yaml    # serve a world over MCP (stdio)
uv run faultbench serve examples/bank/worlds/bank.yaml --http   # ...or over HTTP

Then write your own: docs/WORLDS.md and docs/TESTING.md.

Development

See CLAUDE.md for the session protocol and docs/ for architecture, decisions and status.

uv sync --all-extras
uv run pytest
uv run pytest examples/shop     # the example agent test (needs a model API key)

License

Apache 2.0

Available Tools

4 tools

create_returnD

create_return (custom handler shop_rules.create_return)

ParametersJSON Schema

Name	Required	Description	Default
`order_id`	Yes

TDQS

D1.1/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the full burden of behavioral disclosure. It fails to mention that this likely mutates order state, requires an existing order, or has side effects such as initiating a return process.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short, but it is under-specified rather than genuinely concise. The parenthetical handler name adds no functional value and there is no front-loaded explanation of the tool's purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Even for a tool with a single parameter and no output schema, the description is incomplete. An agent cannot determine what action will occur, what the return value is, or what conditions must be met before calling this tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description adds no meaning to the order_id parameter. The agent only learns that order_id is a required string, but not what it references or how it should be formatted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose1/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'create_return (custom handler shop_rules.create_return)' is a tautology that merely restates the tool name and exposes an internal handler path. It does not state what the tool does, what a return is, or how it differs from sibling tools like issue_refund.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance whatsoever about when to use this tool versus alternatives. No mention of prerequisites, when a return should be created, or when issue_refund might be more appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_orderD

get_order: get on orders

ParametersJSON Schema

Name	Required	Description	Default
`id`	Yes

TDQS

D1.9/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of disclosing behavior. The word 'get' implies a read operation, but it does not state whether this is safe, whether it returns a single order, what fields are included, or how errors are surfaced.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short, but it is under-specified and grammatically awkward ('get on orders'). Conciseness only helps if the content is meaningful; here the brevity sacrifices clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with one parameter and no output schema, the description still should clarify that it retrieves a single order by ID. It does not, leaving the agent to infer behavior from the tool name and siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description adds no meaning beyond the schema. The single required 'id' parameter is present in the schema, but the description does not explain the format, semantics, or why this identifier is needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose2/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'get_order: get on orders' is essentially a tautology that restates the tool name without clarifying what it actually does. It does not specify whether it fetches a single order, multiple orders, or some aggregate view, so an agent cannot distinguish it from list_orders.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this tool versus the sibling tools such as list_orders, create_return, or issue_refund. The description provides no context about relevant scenarios or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

issue_refundD

issue_refund: create on refunds

ParametersJSON Schema

Name	Required	Description	Default
`amount`	No
`order_id`	No
`created_at`	No

TDQS

D1.6/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

There are no annotations, so the description carries the full burden of behavioral disclosure. It indicates a create-like mutation but does not explain side effects, reversibility, idempotency, permissions, or whether a refund modifies an existing order. This is far too thin for a mutating tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely short, but this is under-specification rather than effective conciseness. It front-loads nothing useful and does not earn its place because it adds no operational detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations, no output schema, zero parameter documentation, and a vague one-line description, the tool definition is severely incomplete. An agent cannot determine how to correctly invoke issue_refund or interpret its effects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description adds no meaning to any of the three parameters (amount, order_id, created_at). It does not clarify what amount represents, whether order_id is required logically, or how created_at is used. The schema itself provides only names and types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose2/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'issue_refund: create on refunds' essentially restates the tool name with the verb 'create' and the noun 'refunds'. It does not clearly state what issuing a refund involves, nor does it distinguish this tool from the sibling create_return, which appears to overlap heavily in purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is given about when to use issue_refund versus alternatives like create_return, get_order, or list_orders. There is no mention of prerequisites, required context, or scenarios where a different tool should be chosen.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_ordersD

list_orders: list on orders

ParametersJSON Schema

Name	Required	Description	Default
`customer_id`	No

TDQS

D1.7/5.0

Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavioral traits, but it only restates the action. It does not clarify read-only behavior, pagination, filtering, authorization, or return shape; the word 'list' weakly implies a fetch but no meaningful behavioral context is added.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The text is short but under-specified rather than concise. The phrase 'list on orders' is ungrammatical, and no useful information is effectively front-loaded; a more informative one-liner would not be longer.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter list tool, the description at least indicates it retrieves orders, but it leaves out the meaning of the optional customer_id, how it differs from get_order, and what the response contains. With no annotations or output schema, this is insufficient for reliable tool selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description does not explain customer_id, whether it filters results, or what default null means. The parameter name offers some hint, but the description adds no semantic value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose2/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'list_orders: list on orders' is essentially a restatement of the tool name and adds no definitional value. It says the tool lists orders, but does not specify scope (all orders, by customer, by date) or what distinguishes it from get_order.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance about when to use list_orders versus siblings get_order, create_return, or issue_refund. It does not mention that get_order is likely for a single order or that create_return and issue_refund are for post-order operations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

4 tool updatesv0.1.1Sep 20, 2026
- First observedcreate_return
- First observedget_order
- First observedissue_refund
- First observedlist_orders

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

C2.3/5.0

Scored 2026-09-20 00:55 across 4 tools

Disambiguation5/5

get_order and list_orders are clearly distinguished by cardinality, while create_return and issue_refund target different resources. The action + resource naming leaves little room for an agent to pick the wrong tool.

Naming Consistency5/5

All tools follow a consistent verb_noun snake_case pattern: get_order, list_orders, create_return, issue_refund. The verb choices are clear and the plural/singular usage follows conventional conventions.

Tool Count5/5

Four tools is well-scoped for a focused order/returns/refunds server. Each tool is distinct and purposeful, and the set is small enough for an agent to navigate without unnecessary cognitive load.

Completeness3/5

The server covers order lookup and creating returns/refunds, but lacks read, update, cancel, or verification operations for returns and refunds. This leaves the refund/return lifecycle incomplete and could create dead ends after a return or refund is issued.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

Need Help?

Related Servers

Related MCP Connectors

mockmcp A
io.github.K-Cupples
Hosted MCP endpoint with realistic fake data for prototyping agents. 12 tools, no setup.
Developer Tools Testing & QA Tools
2
WireMock CloudOAuth
io.wiremock
Build, validate, and manage API simulations in WireMock Cloud from MCP-compatible AI agents.
API Testing Testing & QA Tools Developer Tools
1
meok-mcp-test-mcp
io.github.CSOAI-ORG
MEOK MCP Test MCP — golden-file + schema-drift + tool-failure tests for any MCP server. Drop-in
Testing & QA Tools Developer Tools
2
Word Of Clout ACN MCP Server
io.github.wordofclout
Guarded MCP server for agent-readable business truth, provenance, readiness, and discovery.
CRM Customer Data Platforms
2

Related MCP Servers

mcp-mock
Developer Tools Testing & QA Tools
vibeglitch
A
license
Not graded
quality
B
maintenance
A lightweight mock MCP server for local testing and resilience experiments, providing predictable tool responses with simulated latency and errors.
Updated 2 months ago (2026-08-04 12:25 UTC)
1
MIT
datalox-gated-runtime
Agent Orchestration Testing & QA Tools Developer Tools
Oshawott324
A
license
Not graded
quality
A
maintenance
This MCP server provides a stateful, resettable, verifiable API runtime that gates every tool call, enabling agents to run long workflows against provider-shaped environments without live provider write access. It records decisions, side effects, and outcome evidence for replayable, verifiable benchmark runs.
Updated 6 days ago (2026-09-20 14:01 UTC)
Apache 2.0
mcp-explorer
ABeltramo
F
license
Not graded
quality
B
maintenance
A controllable, observable MCP server for dynamically setting tools and intercepting calls in real time, useful as a mock/MITM harness for testing MCP clients.
Updated a month ago (2026-08-14 12:28 UTC)
-
mcpstub
Testing & QA Tools API Testing Developer Tools
jovial-liu
A
license
Not graded
quality
B
maintenance
Serves deterministic MCP protocol fixtures generated from tool contracts, enabling testing of MCP clients, gateways, and agent harnesses without live credentials or external services.
Updated a month ago (2026-08-23 10:27 UTC)
MIT