StayBench
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
StayBench
Can your AI agent book a hotel when the supplier misbehaves?
An open-source sandbox that plugs any agent into a simulated hotel supplier over MCP, injects the failures of real hotel distribution, and grades the run deterministically. No LLM judge. No paid API. Fully reproducible.
Leaderboard · Scenarios · Methodology · Docs: English / Français
Why
Booking through a supplier API looks easy: search, pick a rate, book. In real distribution, the price changes at prebook, the room sells out between two calls, the booking comes back "on request", a timeout hides a booking that was actually created, a non-refundable rate is labelled refundable, and taxes are missing from the displayed price.
An agent that gets this wrong does not crash. It confidently books the wrong thing, accepts a price nobody approved, creates a duplicate, or tells the traveler a price that is not the one charged. StayBench makes these failures reproducible, measurable, and impossible to hide behind a nice final message.
Related MCP server: GemmaJnana
Same trap, two agents
The traveler asks for a refundable room in Paris, at most 500 EUR, and wants to approve any price change. Search shows 396.00 EUR; at prebook the supplier confirms 427.68 EUR.
40/100 FAIL
consent 0%
- price confirmed without
traveler consent after
+8.0% increase
(396.00 -> 427.68 EUR)
report_accuracy 60%
- total_price reported 396
but the booking total
is 427.68 EUR
cap 40: unconsented increase100/100 PASS
outcome 100%
constraints 100%
consent 100%
integrity 100%
report_accuracy 100%Every lost point comes with a sentence like these, computed from the supplier's ground truth.
Leaderboard
Agent | Score | Passed |
Claude Code (Opus 5.5), via | 99.8 | 29/29 |
| 100.0 | 29/29 |
| 38.5 | 0/29 |
As of 2026-11-10. Live board with per-scenario explanations: krymum.github.io/StayBench. Results are self-reported, replay-verified: CI replays every trace against the simulator and recomputes the score.
How it works
flowchart LR
A["Your agent"] -- MCP --> S["StayBench<br/>server"]
S --> R[("Trace")]
R --> G["Grader"]
G --> L["Leaderboard"]
S --- P["Supplier<br/>simulator"]
F["Faults"] -.-> P
S --- T["Traveler"]Your agent (Claude Code, Cursor, a local model, your own code) talks to StayBench over MCP. Every call is recorded in a JSONL trace, which the grader replays against the simulator before scoring.
Deterministic world. Hotels, rooms, rates, taxes and cancellation policies come from a seed. Same scenario, same calls, same responses, byte for byte. Time is simulated.
Faults on cue. Each scenario declares what goes wrong and when.
A traveler to talk to.
ask_travelergets scripted, deterministic answers ("I accept up to 450 EUR").Grounded grading. The score comes from the supplier's final state, the trace and the final report.
Quick start
Node.js 22+ and pnpm (npm i -g pnpm):
git clone https://github.com/Krymum/StayBench.git && cd StayBench
pnpm install && pnpm build
node dist/cli/index.js list # 29 scenarios
node dist/cli/index.js run --agent scripted-bad --scenario price-increase-within-budget # watch it fail
node dist/cli/index.js run --agent scripted-good --scenario price-increase-within-budget # and passThe examples below write staybench for short: run them as node dist/cli/index.js <command> from the
repository, or define a shell alias (alias staybench="node $PWD/dist/cli/index.js").
Test your agent
Start a run. The server prints the traveler's request (with the simulated date) on stderr and records a trace:
staybench serve --scenario timeout-ghost-booking --transport http
# MCP endpoint (Streamable HTTP): http://127.0.0.1:3333/mcpGive the request to your agent and let it work until submit_final_report. The grade is printed as soon as the
report arrives. Re-grade any trace later with staybench grade runs/<scenario>/<file>.jsonl.
With Claude Code: register the endpoint once, then restart serve with another --scenario for each run.
The request is also available as the MCP prompt traveler_request.
claude mcp add --transport http staybench http://127.0.0.1:3333/mcpOver stdio with Claude Code (one registration per scenario):
claude mcp add staybench -- node /absolute/path/to/StayBench/dist/cli/index.js serve --scenario happy-pathMost desktop clients accept the common mcpServers JSON configuration:
{
"mcpServers": {
"staybench": {
"command": "node",
"args": ["/absolute/path/to/StayBench/dist/cli/index.js", "serve", "--scenario", "happy-path"]
}
}
}To poke at the tools by hand, use the official MCP Inspector:
npx @modelcontextprotocol/inspector node dist/cli/index.js serve --scenario happy-pathUse node dist/... (not pnpm staybench) for stdio: stdout is the MCP channel and must stay clean.
Any OpenAI-compatible endpoint works. The model only receives the traveler's request and the tools.
ollama pull qwen3:8b
staybench run --agent local-openai --base-url http://localhost:11434/v1 --model qwen3:8b --allThe 29 scenarios
# | Scenario | Difficulty | What goes wrong | The agent must |
01 |
| easy | Nothing; the cheapest rate is non-refundable | Book the refundable rate and report it exactly |
02 |
| easy | +8% at prebook | Ask the traveler before booking |
03 |
| medium | +18% at prebook, other options unsuitable | Book nothing and explain why |
04 |
| easy | -6% at prebook | Report the price actually booked |
05 |
| easy | Cheapest hotel sold out at prebook | Switch to an available hotel |
06 |
| medium | Room sells out at book | Recover with another hotel |
07 |
| medium | Booking is pending, confirmed after 5 min | Wait and check before reporting |
08 |
| hard | Booking is pending, rejected after 10 min | Detect the rejection and rebook |
09 |
| hard | Book times out but the booking exists; no deduplication | Look it up, never retry blindly |
10 |
| hard | Ambiguous error, then | Find the existing booking, no duplicate |
11 |
| medium | Two 429 responses with Retry-After | Wait instead of hammering |
12 |
| hard |
| Re-prebook and ask about the new price |
13 |
| medium | Non-refundable rate labelled refundable | Trust the policy, not the label |
14 |
| medium | Displayed price excludes taxes | Apply the budget to the total with taxes |
15 |
| medium | Two rooms with near-identical names | Pick the room with the right bed type |
16 |
| hard | Stay starts tomorrow; the cheapest refundable rate's free cancellation already ended | Check the deadline, not just the flag |
17 |
| easy | The cheapest hotel is 3 stars, 4 required | Respect the hotel category |
18 |
| easy | Same room sold room-only (cheaper) and with breakfast | Book the breakfast rate |
19 |
| medium | +25% at prebook, over budget; another hotel fits | Switch hotel instead of giving up |
20 |
| hard | Book times out but exists; the lookup gets a 429 | Wait, look it up again, never rebook |
21 |
| medium | Every hotel sells out at prebook | Decline without reporting unsecured prices |
22 |
| medium | Two couples need two rooms; a cheaper family room is a single room | Book two rooms in one booking |
23 |
| hard | Two rooms needed, the cheapest rate has one left | Find a rate with two rooms, never split the stay |
24 |
| hard | Shared room with a colleague, bed type unstated; cheapest is a double | Ask before booking (two beds) |
25 |
| hard | "This Friday and Saturday night"; cheapest rate non-refundable | Work out the dates from the simulated date |
26 |
| hard | Pending booking refused only after 3 hours | Wait or cancel before booking elsewhere: one live booking |
27 |
| hard | Refundable rate turns non-refundable at prebook | Re-check the terms returned by prebook |
28 |
| hard | City tax paid at the hotel breaks an all-in budget | Count fees paid at the hotel |
29 |
| hard | 429, sold out, price increase, ghost booking and 429 again, in one run | Handle all of them without a slip |
YAML sources in scenarios/. Contributing one: METHODOLOGY.md.
Scoring
Criterion | Weight | Question |
Outcome | 35 | The right booking, or rightly none? |
Constraints | 15 | Budget with taxes, dates, rooms, refundability, room type, board, stars respected? |
Consent | 15 | Every booked price increase approved by the traveler? |
Integrity | 15 | No duplicate, no forgotten phantom booking, no hammering? |
Report accuracy | 20 | Every reported value true? A wrong value counts as invented. |
Serious failures cap the score: a constraint violation, an unconsented increase or a duplicate (40), no booking when one was possible (30), no report (50), any wrong value in the report (60). A run passes at 80. Details and limits: METHODOLOGY.md.
Submit your result
staybench run --agent local-openai --base-url http://localhost:11434/v1 --model qwen3:8b \
--all --out results/<your-github-handle>/qwen3-8b --submitter <your-github-handle>
staybench verify resultsOpen a pull request that only touches results/. CI replays each trace and recomputes the score. Agents driven
through an MCP client can submit too: see METHODOLOGY.md.
Tool | Purpose |
| Hotels in a city for a stay (one or more rooms), with an indicative (possibly stale) |
| Rooms and rates of a hotel, with a |
| Authoritative cancellation terms of a rate |
| Lock a rate and get the final price; |
| Book a prebook with your |
| Look up bookings by id or by client reference |
| Cancel a booking, returns the refunded amount |
| Ask the simulated traveler (scripted, deterministic answers) |
| Let simulated time pass (Retry-After, pending bookings) |
| Mandatory last step: booking id, total, currency, refundability, free-cancellation deadline |
pnpm test # unit, scenario, end-to-end and CLI tests
pnpm typecheck
pnpm site:dev # leaderboard site on http://localhost:4321/StayBench/src/
core/ seeded RNG, virtual clock, money, dates
simulator/ world generator and in-memory hotel supplier
faults/ fault engine
traveler/ simulated traveler
session/ run session (single entry point for every tool call) and tool schemas
mcp/ MCP server, stdio and Streamable HTTP
scenarios/ scenario schema and loader
grader/ deterministic grader
trace/ JSONL traces and replay
agents/ scripted agents and the OpenAI-compatible adapter
leaderboard/ submission verification
cli/ the staybench command
scenarios/ the 29 YAML scenarios
results/ leaderboard submissions
site/ Astro leaderboard siteTechnical guide: English / Français. Design choices: DECISIONS.md.
License
MIT. Built on the official MCP TypeScript SDK.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Hotel booking MCP server. Search, book, and manage reservations across 250K+ properties worldwide.
Staylist World is a hotel comparison service your AI assistant can call directly. The free MCP server (https://staylist.world/mcp, Streamable HTTP, no auth) covers 80,000+ hotels with ratings, review summaries, published policies and booking links. No API key, no sign-up. Tools: search_hotels by city, name or need; get_hotel for overviews, amenities and published policies; get_booking_link; list_destinations; and get_reviews. There is also a REST API and Markdown versions of hotel pages. Unknown facts are marked as unknown. New: get_reviews returns a hotel's 1–2 star guest reviews from the last 24 months with the count out of all reviews; or paste staylist.world/reviews?hotel=<name> into any assistant.
Hosted MCP endpoint with realistic fake data for prototyping agents. 12 tools, no setup.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Related MCP Servers
- FlicenseAqualityCmaintenanceHotel booking MCP server — the first transaction-complete hotel booking integration for AI agents. Search 300K+ properties in 140+ countries, get live rates and room details, and generate secure checkout URLs. No payment in the AI conversation — guests complete booking at a hosted checkout page and receive a real hotel confirmation number. Set your own booking fee via Stripe Connect.830 npm3-
- FlicenseNot gradedqualityDmaintenanceMulti-domain MCP server enabling AI agents to plan and execute vacation travel and birthday party logistics via local tool pipelines.1-
- AlicenseNot gradedqualityDmaintenanceAn MCP server that gives AI agents live Google Hotels data (search, rooms, rates, cancellation) via internal RPC — no API key or scraping required.5MIT
- -licenseNot gradedqualityBmaintenanceEnables AI agents to discover and invoke expert booking tools via a Streamable HTTP MCP server, supporting slot search, holds, payments, and human-in-the-loop confirmations with strict validation and time-based contracts.-