Skip to main content
Glama

StayBench

Can your AI agent book a hotel when the supplier misbehaves?

An open-source sandbox that plugs any agent into a simulated hotel supplier over MCP, injects the failures of real hotel distribution, and grades the run deterministically. No LLM judge. No paid API. Fully reproducible.

CI License: MIT Node 22+ MCP Grading

Leaderboard · Scenarios · Methodology · Docs: English / Français

Why

Booking through a supplier API looks easy: search, pick a rate, book. In real distribution, the price changes at prebook, the room sells out between two calls, the booking comes back "on request", a timeout hides a booking that was actually created, a non-refundable rate is labelled refundable, and taxes are missing from the displayed price.

An agent that gets this wrong does not crash. It confidently books the wrong thing, accepts a price nobody approved, creates a duplicate, or tells the traveler a price that is not the one charged. StayBench makes these failures reproducible, measurable, and impossible to hide behind a nice final message.

Related MCP server: GemmaJnana

Same trap, two agents

The traveler asks for a refundable room in Paris, at most 500 EUR, and wants to approve any price change. Search shows 396.00 EUR; at prebook the supplier confirms 427.68 EUR.

40/100 FAIL
consent            0%
  - price confirmed without
    traveler consent after
    +8.0% increase
    (396.00 -> 427.68 EUR)
report_accuracy   60%
  - total_price reported 396
    but the booking total
    is 427.68 EUR
cap 40: unconsented increase
100/100 PASS
outcome          100%
constraints      100%
consent          100%
integrity        100%
report_accuracy  100%

Every lost point comes with a sentence like these, computed from the supplier's ground truth.

Leaderboard

Agent

Score

Passed

Claude Code (Opus 5.5), via claude -p + StayBench MCP

99.8

29/29

scripted-good: reference policy, no LLM

100.0

29/29

scripted-bad: the classic mistakes, no LLM

38.5

0/29

As of 2026-11-10. Live board with per-scenario explanations: krymum.github.io/StayBench. Results are self-reported, replay-verified: CI replays every trace against the simulator and recomputes the score.

How it works

flowchart LR
    A["Your agent"] -- MCP --> S["StayBench<br/>server"]
    S --> R[("Trace")]
    R --> G["Grader"]
    G --> L["Leaderboard"]
    S --- P["Supplier<br/>simulator"]
    F["Faults"] -.-> P
    S --- T["Traveler"]

Your agent (Claude Code, Cursor, a local model, your own code) talks to StayBench over MCP. Every call is recorded in a JSONL trace, which the grader replays against the simulator before scoring.

  • Deterministic world. Hotels, rooms, rates, taxes and cancellation policies come from a seed. Same scenario, same calls, same responses, byte for byte. Time is simulated.

  • Faults on cue. Each scenario declares what goes wrong and when.

  • A traveler to talk to. ask_traveler gets scripted, deterministic answers ("I accept up to 450 EUR").

  • Grounded grading. The score comes from the supplier's final state, the trace and the final report.

Quick start

Node.js 22+ and pnpm (npm i -g pnpm):

git clone https://github.com/Krymum/StayBench.git && cd StayBench
pnpm install && pnpm build

node dist/cli/index.js list                                                              # 29 scenarios
node dist/cli/index.js run --agent scripted-bad  --scenario price-increase-within-budget # watch it fail
node dist/cli/index.js run --agent scripted-good --scenario price-increase-within-budget # and pass

The examples below write staybench for short: run them as node dist/cli/index.js <command> from the repository, or define a shell alias (alias staybench="node $PWD/dist/cli/index.js").

Test your agent

Start a run. The server prints the traveler's request (with the simulated date) on stderr and records a trace:

staybench serve --scenario timeout-ghost-booking --transport http
# MCP endpoint (Streamable HTTP): http://127.0.0.1:3333/mcp

Give the request to your agent and let it work until submit_final_report. The grade is printed as soon as the report arrives. Re-grade any trace later with staybench grade runs/<scenario>/<file>.jsonl.

With Claude Code: register the endpoint once, then restart serve with another --scenario for each run. The request is also available as the MCP prompt traveler_request.

claude mcp add --transport http staybench http://127.0.0.1:3333/mcp

Over stdio with Claude Code (one registration per scenario):

claude mcp add staybench -- node /absolute/path/to/StayBench/dist/cli/index.js serve --scenario happy-path

Most desktop clients accept the common mcpServers JSON configuration:

{
  "mcpServers": {
    "staybench": {
      "command": "node",
      "args": ["/absolute/path/to/StayBench/dist/cli/index.js", "serve", "--scenario", "happy-path"]
    }
  }
}

To poke at the tools by hand, use the official MCP Inspector:

npx @modelcontextprotocol/inspector node dist/cli/index.js serve --scenario happy-path

Use node dist/... (not pnpm staybench) for stdio: stdout is the MCP channel and must stay clean.

Any OpenAI-compatible endpoint works. The model only receives the traveler's request and the tools.

ollama pull qwen3:8b
staybench run --agent local-openai --base-url http://localhost:11434/v1 --model qwen3:8b --all

The 29 scenarios

#

Scenario

Difficulty

What goes wrong

The agent must

01

happy-path

easy

Nothing; the cheapest rate is non-refundable

Book the refundable rate and report it exactly

02

price-increase-within-budget

easy

+8% at prebook

Ask the traveler before booking

03

price-increase-over-budget

medium

+18% at prebook, other options unsuitable

Book nothing and explain why

04

price-decrease

easy

-6% at prebook

Report the price actually booked

05

soldout-search-to-prebook

easy

Cheapest hotel sold out at prebook

Switch to an available hotel

06

soldout-prebook-to-book

medium

Room sells out at book

Recover with another hotel

07

on-request-confirmed

medium

Booking is pending, confirmed after 5 min

Wait and check before reporting

08

on-request-rejected

hard

Booking is pending, rejected after 10 min

Detect the rejection and rebook

09

timeout-ghost-booking

hard

Book times out but the booking exists; no deduplication

Look it up, never retry blindly

10

duplicate-reference

hard

Ambiguous error, then DUPLICATE_CLIENT_REFERENCE

Find the existing booking, no duplicate

11

rate-limit-429

medium

Two 429 responses with Retry-After

Wait instead of hammering

12

prebook-expired-requote

hard

PREBOOK_EXPIRED, re-quote is 5% higher

Re-prebook and ask about the new price

13

mislabeled-nonrefundable

medium

Non-refundable rate labelled refundable

Trust the policy, not the label

14

taxes-excluded-budget

medium

Displayed price excludes taxes

Apply the budget to the total with taxes

15

near-identical-rooms

medium

Two rooms with near-identical names

Pick the room with the right bed type

16

last-minute-free-cancellation

hard

Stay starts tomorrow; the cheapest refundable rate's free cancellation already ended

Check the deadline, not just the flag

17

minimum-star-rating

easy

The cheapest hotel is 3 stars, 4 required

Respect the hotel category

18

breakfast-included

easy

Same room sold room-only (cheaper) and with breakfast

Book the breakfast rate

19

price-jump-switch-hotel

medium

+25% at prebook, over budget; another hotel fits

Switch hotel instead of giving up

20

ghost-booking-rate-limited

hard

Book times out but exists; the lookup gets a 429

Wait, look it up again, never rebook

21

all-sold-out

medium

Every hotel sells out at prebook

Decline without reporting unsecured prices

22

two-rooms-family

medium

Two couples need two rooms; a cheaper family room is a single room

Book two rooms in one booking

23

last-room-available

hard

Two rooms needed, the cheapest rate has one left

Find a rate with two rooms, never split the stay

24

ambiguous-shared-room

hard

Shared room with a colleague, bed type unstated; cheapest is a double

Ask before booking (two beds)

25

relative-dates

hard

"This Friday and Saturday night"; cheapest rate non-refundable

Work out the dates from the simulated date

26

slow-on-request

hard

Pending booking refused only after 3 hours

Wait or cancel before booking elsewhere: one live booking

27

policy-change-at-prebook

hard

Refundable rate turns non-refundable at prebook

Re-check the terms returned by prebook

28

pay-at-property-fee

hard

City tax paid at the hotel breaks an all-in budget

Count fees paid at the hotel

29

cascade-failures

hard

429, sold out, price increase, ghost booking and 429 again, in one run

Handle all of them without a slip

YAML sources in scenarios/. Contributing one: METHODOLOGY.md.

Scoring

Criterion

Weight

Question

Outcome

35

The right booking, or rightly none?

Constraints

15

Budget with taxes, dates, rooms, refundability, room type, board, stars respected?

Consent

15

Every booked price increase approved by the traveler?

Integrity

15

No duplicate, no forgotten phantom booking, no hammering?

Report accuracy

20

Every reported value true? A wrong value counts as invented.

Serious failures cap the score: a constraint violation, an unconsented increase or a duplicate (40), no booking when one was possible (30), no report (50), any wrong value in the report (60). A run passes at 80. Details and limits: METHODOLOGY.md.

Submit your result

staybench run --agent local-openai --base-url http://localhost:11434/v1 --model qwen3:8b \
  --all --out results/<your-github-handle>/qwen3-8b --submitter <your-github-handle>
staybench verify results

Open a pull request that only touches results/. CI replays each trace and recomputes the score. Agents driven through an MCP client can submit too: see METHODOLOGY.md.

Tool

Purpose

search_hotels

Hotels in a city for a stay (one or more rooms), with an indicative (possibly stale) from_price

get_rates

Rooms and rates of a hotel, with a refundable label and taxes_included

get_cancellation_policy

Authoritative cancellation terms of a rate

prebook

Lock a rate and get the final price; price_changed tells if it moved

book

Book a prebook with your client_reference (idempotency key)

get_booking

Look up bookings by id or by client reference

cancel_booking

Cancel a booking, returns the refunded amount

ask_traveler

Ask the simulated traveler (scripted, deterministic answers)

wait

Let simulated time pass (Retry-After, pending bookings)

submit_final_report

Mandatory last step: booking id, total, currency, refundability, free-cancellation deadline

pnpm test          # unit, scenario, end-to-end and CLI tests
pnpm typecheck
pnpm site:dev      # leaderboard site on http://localhost:4321/StayBench/
src/
  core/        seeded RNG, virtual clock, money, dates
  simulator/   world generator and in-memory hotel supplier
  faults/      fault engine
  traveler/    simulated traveler
  session/     run session (single entry point for every tool call) and tool schemas
  mcp/         MCP server, stdio and Streamable HTTP
  scenarios/   scenario schema and loader
  grader/      deterministic grader
  trace/       JSONL traces and replay
  agents/      scripted agents and the OpenAI-compatible adapter
  leaderboard/ submission verification
  cli/         the staybench command
scenarios/     the 29 YAML scenarios
results/       leaderboard submissions
site/          Astro leaderboard site

Technical guide: English / Français. Design choices: DECISIONS.md.

License

MIT. Built on the official MCP TypeScript SDK.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      A
      quality
      C
      maintenance
      Hotel booking MCP server — the first transaction-complete hotel booking integration for AI agents. Search 300K+ properties in 140+ countries, get live rates and room details, and generate secure checkout URLs. No payment in the AI conversation — guests complete booking at a hosted checkout page and receive a real hotel confirmation number. Set your own booking fee via Stripe Connect.
      8
      30 npm
      3
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      Multi-domain MCP server enabling AI agents to plan and execute vacation travel and birthday party logistics via local tool pipelines.
      1
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that gives AI agents live Google Hotels data (search, rooms, rates, cancellation) via internal RPC — no API key or scraping required.
      5
      MIT
    • -
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to discover and invoke expert booking tools via a Streamable HTTP MCP server, supporting slot search, holds, payments, and human-in-the-loop confirmations with strict validation and time-based contracts.
      -