MarketMirror
This server (MarketMirror) lets you run simulated AI market research — synthetic buyers reacting to ads, prices, emails, pitches and real websites — and turn the results into an evidence-backed decision.
Setup & diagnostics: check install/keys/backend (
doctor), install the engine (setup), list/configure/discover model providers (provider_setup), start the backend (start_backend).Understand the product: build a product profile from docs, landing page or ads export (
understand), collect cited public feedback (research), convert ad exports into seeds/benchmarks (ads_import), and draft a campaign plan with runnable test specs (campaign_plan).Build and run simulations: generate a ready scenario with simulated buyers (
quickstart), run the full pipeline (feed test, social rounds, survey, panel, report) (market_test), or test specific variants — ad, message, price, offer, naming, landing, launch, pitch, email, creator post, channel mix (run_test).Specialized tests: inbox/open/click tests (
email_test), sales pitch + rebuttal rehearsal with lead scores (pitch), synthetic visitors browsing a real website with screenshots and DOM checks (website_test), and multi-scenario × multi-seed comparisons with a defended conclusion (study).Live grounding: inject news, competitor moves or price changes into a running simulation (
push_context).Inspect results: key decisions, verdicts, objections, pricing, feed metrics and ROI (
results), list past runs (list_runs), poll background jobs (job_status), serve the live dashboard (start_live).Validate against reality: normalize real exports (
actuals_import), score predictions against real ad results (backtest), run held-out backtests (holdout_backtest), calibrate surveys against real answers (calibrate), forecast with intervals (predict), and get prioritized next-test ideas (ideas).Advanced: run any underlying CLI command directly (
cli).
Simulates Facebook feed environments for ad testing, placing ads among ordinary posts so simulated buyers can stop, ignore, click, comment on, or hide them.
Simulates Instagram feed environments for ad testing, placing ads among ordinary posts so simulated buyers can stop, ignore, click, comment on, or hide them.
Simulates Reddit feed environments for ad testing, placing ads among ordinary posts so simulated buyers can stop, ignore, click, comment on, or hide them.
Simulates TikTok feed environments for ad testing, placing ads among ordinary posts so simulated buyers can stop, ignore, click, comment on, or hide them.
MarketMirror: AI Market Research & Marketing Simulation
Try your ads, prices, and sales pitch with simulated buyers before spending on a real campaign.
English · 简体中文
Open source and built for AI agents. Use it from Claude Code, Codex, Cursor, Gemini, or any MCP-compatible agent. Install the plugin, ask in plain words ("test these three ads on young parents in India"), and your agent builds the simulated buyers, runs the test, and shows you the dashboard. Self-hosted: it runs on your computer; only the model calls (to the provider you choose) and optional public research leave it.
▶ Watch the 60-second launch film (with captions) · vertical 9:16 cut · made with HyperFrames from launch-video/
Open the live demo: the full dashboard from a 2,999-buyer simulated run, in your browser, no install. (Fitloop is an invented app; every number is simulated.)
If MarketMirror saves you one bad launch, a ⭐ helps other people find it.
Meet the buyers · What you can test · Screenshots · How it works · Example results · Get started · Documentation
What is MarketMirror?
MarketMirror gives you a practice audience for your next marketing decision. Describe your product and the people you want to reach. It creates AI-generated people who can react to your ads, compare prices, ask questions, raise concerns, and respond to a sales pitch.
You get a report explaining what interested them, what put them off, what to change, and what to test with real customers next. You can also watch the simulation and explore the results in a dashboard.
For example: you are launching a fitness app. Should the ad lead with “15 minutes a day” or “less than one gym visit”? Should the free trial ask for a payment card? MarketMirror lets you rehearse those choices before a live campaign.
The buyers are simulated. Their responses are generated by AI, so use them to develop and compare ideas. Confirm promising findings with real customers and small live experiments.
Related MCP server: mock-llm-mcp
Who is it for?
If you are… | Use MarketMirror to… |
A founder or small business owner | Explore how different people might understand your product, price, or launch offer. |
A marketer or agency | Compare ads, headlines, emails, and audiences before choosing a live test. |
A sales team | Practice a pitch, discover likely concerns, and try clearer answers. |
A product or research team | Explore buying decisions and website journeys, then plan customer research. |
A developer or AI agent builder | Run the same workflows through commands, an API, a Python library, or connected AI assistants. |
Meet the simulated buyers
A simulated buyer is an AI-generated character representing someone in your intended audience. The character has a background and practical reasons for accepting, questioning, or rejecting an offer.
You choose the groups you want to explore—for example, students, working parents, and young professionals in selected countries. MarketMirror creates different people within those groups and can ask them about their lives before showing them your marketing.
What a buyer can have | Why it matters in a test |
A background | Age, location, work, and life stage provide context for their response. |
A budget and priorities | An offer may interest them but still compete with more urgent expenses. |
A phone and internet connection | A heavy page or video may be inconvenient on their device or data plan. |
People they consult | A partner, family member, or community may influence a purchase. |
Trust concerns | Hidden fees, unclear claims, or a previous bad experience may make them hesitate. |
A buying decision card | A summary records what could persuade them, what could stop them, and who else matters. |
An illustrative buyer: a busy parent who wants short home workouts, uses a budget phone, and worries about surprise charges. They might like a low monthly price but reject a “free” trial that immediately asks for a payment card. Another buyer may care more about workout quality than price. These are invented examples, not findings from real customers.
In tests within the same run, buyers can remember earlier experiences. Someone who disliked an ad can carry that reaction into a later survey or pitch. Separate comparison scenarios keep their own histories.
Detailed characters help you explore different viewpoints. They do not make the generated answers representative of a real population.
What you can test
Your question | What MarketMirror does | What you can learn from the simulation |
Which ad gets attention? | Places your ads among ordinary posts in simulated Facebook, Instagram, Reddit, TikTok, LinkedIn, or X feeds. | Which ads buyers stop at, ignore, click, comment on, or hide. |
Which headline or message is clearer? | Shows different versions to different buyers and asks for their reactions. | Which wording draws interest and which creates confusion. |
Does the price or offer feel right? | Asks price questions and compares offers or pricing tiers. | Where price becomes a concern and which terms need explaining. |
What might happen around a launch? | Lets buyers post, reply, and react in a simulated social conversation with scheduled brand posts and events. | Questions, objections, and themes that emerge in the discussion. |
Will an email interest them? | Tests subject lines, email content, or a sequence in simulated inboxes. | Generated opening, reading, and response choices. No emails are sent. |
How does our sales pitch land? | Presents the pitch, collects concerns, tries your answers, and asks again. | Which concerns persist and whether your response helps or makes things worse. |
Where does our website lose people? | Sends simulated visitors through a real website and records their actions. | Where they hesitate, leave, or complete a checked task such as adding to cart. |
Which audience should we investigate first? | Compares responses across the groups and countries you selected. | Differences worth following up with real customer research. |
What should we test next? | Uses the run and, when supplied, real campaign exports to suggest follow-up experiments. | A practical starting point for the next live test. |
Do the simulations match our results? | Compares paired simulated and observed outcomes. | How well the simulation held up on those experiments and where it needs adjustment. |
You can repeat a comparison to see whether its result changes between runs. You can also use different AI providers and inspect where they disagree. Both help reveal uncertainty within the simulation.
The process at a glance

Describe your product. Provide a brief, documents, or a website: what you sell, who it helps, what it costs, and what supports your claims. You can also bring linked public feedback or campaign exports.
Choose your audience. Describe the groups and locations you want to explore. MarketMirror creates the simulated people. Review their profiles before continuing.
Give them something to react to. Add your ads, messages, prices, emails, pitch, or website task. Choose the tests that answer your question.
Read the recommendation. Explore the responses, concerns, comparisons, and suggested actions in the report and dashboard.
Try a small real experiment. Use actual customer responses or campaign outcomes to check what held up. Import those outcomes for comparison when you have suitable data.
The written report puts one recommendation at the top:
Recommendation | How to use it |
Go | Take the promising idea into a small real test. |
Iterate | Change the weak parts and test again. |
No-go | Reconsider this version before putting money behind it. |
These recommendations summarize simulation signals. A “go” does not guarantee sales.
What you get
Actual product screenshot. Fitloop is an invented example app; all displayed buyer responses and performance figures are simulated. The illustrated banner and workflow above are conceptual artwork.
A decision and next actions: what to keep, what to change, and what to test with real people.
Buyer responses: who was simulated, what they said, and their main concerns.
Comparisons: differences between ads, messages, offers, audiences, and AI providers when those tests were run.
A shareable report: a saved HTML dashboard that opens without a server, works offline, and includes tables behind its charts.
Downloadable results: report and data exports in Markdown, JSON, CSV, and PDF.
A live view: watch progress, buyer activity, and social posts as a run happens.
The full screenshot gallery and detailed explanations follow below.
The dashboard
Each run produces one self-contained HTML dashboard: no server, no external requests, light and dark themes, and phone layouts. A sidebar groups its sections into overview, buyers, tests, evidence and the simulation itself; every chart has a table behind it, and the report exports to Markdown, JSON, CSV and PDF. In runs with more than 500 agents, the agents table lists the 500 most active and says so; everyone is still in the people cards and the CSV downloads. The screenshots below come from the 3,000-person Fitloop market (people written by Claude, Codex and Gemini), except the study page (the offer study) and the live control room (the 1,000-person run). scripts/readme_screenshots.py regenerates them from any run's dashboard.html with your installed Chrome.
Fitloop, its audiences and its pilot figures are invented; every number in these screenshots is simulated, not observed campaign performance. Report generation checks numbers against the run's fact sheet and falls back to a data-only report if the model fails. Those checks establish consistency with saved outputs, not real-world accuracy.
How it works
Stage | Input → output |
Understand | Product brief, documents, or website → product profile and knowledge graph |
Build | Audience segments and public feedback → simulated buyers with interviews and decision cards |
Test | Ads, messages, prices, posts, or pitches → reactions, objections, and comparisons |
Decide | Saved simulation outputs → recommendation, report, and dashboard |
Validate | Observed campaign or survey outcomes → backtests, calibration, and the next experiment |
One command, market-test, runs the configured pipeline stages from a single config file. Individual tests can also run on their own. Everything a step produces is saved in the run folder, so a stopped run resumes where it stopped.
1. Understand the product
MarketMirror reads your brief, documents or website and writes a product profile: who it is for, the offer, the positioning, the proof, the competitors, and what is still unknown. Every later prompt reads this profile, so the simulated people react to your actual product rather than a vague idea of it.
Your documents also become a small knowledge graph (stored in local SQLite, searched with local embeddings, no extra service). Organisations and public figures in it can join the simulation as accounts that post.
Optionally, research collects what real people already say: Hacker News threads, country-specific App Store reviews, Reddit results and pages you supply, each with its link. Those quotes are shown to the people who write the simulated buyers, so the market starts from real wording.
2. Build the market
You describe the audiences in a small spec: "Students in India, 18-24, 500 people; parents in the Philippines, 30-45, 400 people", plus countries, cities and communities such as a WhatsApp running group or a subreddit. Then, for every person:
flowchart LR
S["Slot<br/>age · gender · city<br/>income · life stage"] --> P["Persona<br/>150-250 words"]
P --> G["Stereotype guard<br/>no two alike in job<br/>and city"]
G --> I["Life interview<br/>12 questions +<br/>first-person summary"]
I --> F["Follow-ups<br/>2 probes · 3 product<br/>questions · decision card"]Slots spread each audience evenly over ages, genders, cities (big and small), incomes and life stages, so you never get 500 copies of the same 24-year-old.
The stereotype guard makes sure no two people in an audience share both a job and a city (it moves one to another city or varies the role, in code, with no model call) and flags any audience whose people came out too alike.
The life interview asks what a researcher would: Where did you grow up? How is money this month, and what do you cut first? What phone do you use, and what is your data plan like? Tell me about a time you got scammed. Who do you ask before spending real money? What annoys you most in ads? The answers become a first-person summary in the person's own voice and language.
The decision card is how this person buys: who they ask, how fast they decide, what triggers a yes, what stops them, their dealbreakers, the trust cues they look for, their monthly spare money, and where what they say differs from what they do.
Communities give people somewhere to talk: each person joins a few of the groups you named, and they follow each other there.
One person from the 3,000-person run, exactly as the model wrote him (abridged):
Who | Yash Malhotra, 42, Pune. Owner-driver of an app cab; runs a mobile-accessories stall at weekends. Lower-middle income, Realme 9 Pro, Jio prepaid ₹349 (2 GB a day). Member of Working Moms & Dads India. |
In his words | "Mi Yash Malhotra, 42, Pune madhe cab chalvto ani weekend la mobile accessories stall lavto… crypto course madhe ₹6,000 gelele mhanun reviews bghto ani bayko aai la vicharto." (He lost ₹6,000 to a crypto course, so he reads reviews and asks his wife and mother.) |
Asks before buying | Pooja (his wife), his mother, and Santosh at the stall |
Says yes to | free trial, cheap price, family plan, a Hindi coach |
Walks away from | over-promising ads, long unskippable ads, a card needed upfront; hidden charges |
Trusts | reviews with 4+ stars, clear prices, family opinion |
Says vs does | Says he would do 15 minutes during waiting time, but his shifts run long and he is tired, so he likely will not follow through. |
The people keep a memory for the whole run: someone who hid your ad or ignored your email carries that into the survey and the pitch. With several model providers, each provider writes its own share of the people and plays them from start to finish (mixed markets).
3. Put them in the world
Ads in realistic feeds (the feed test). Each person scrolls a feed for each platform you test (Facebook, Instagram, Reddit, TikTok, LinkedIn, X): a handful of local organic posts with one of your ads placed where that platform puts its sponsored posts. When you test several ads, each person sees exactly one, assigned at random, so the comparison is a split test. For every post they report what they did: scrolled past, stopped and for how long, watched, reacted, commented, shared, saved, clicked or hid it, plus a one-line thought. Platform playbooks hold typical base rates (most people scroll past most ads), which keeps simulated people from being unrealistically positive. You get ads-manager numbers per ad and per audience (thumb-stop rate, CTR, what happened after the click), and with --seeds 3 a range instead of one number.
Social rounds. The people then live on Reddit- and X-like platforms for a few rounds, powered by the OASIS engine with its recommendation feed. Your brand posts and your launch calendar (a price drop on day 3, an influencer post on day 5) arrive in the round you schedule them, and you can push live news into a running simulation ("a competitor just cut its price"). People post, reply, like, follow and argue in their own voices. The dashboard's Social feed shows every post by round, audience and country, with the replies to your brand posts and their sentiment.
4. Ask them
The survey. Everyone answers in their own words: would they try it, how interested are they, do they trust it, would they recommend it. Free text is turned into 1-5 rating distributions with semantic similarity rating (SSR): each answer is compared with five anchor statements using local embeddings, with no extra model calls. Asked for a number directly, language models give narrow, unrealistic rating distributions; rating their own words gives distributions much closer to real survey panels, as the SSR paper reports. The survey can also:
split-test your messages, with each person randomly shown one;
ask four price questions and draw the acceptable price range and sweet spot (Van Westendorp);
re-ask a sample "a simulated week later" and report how many gave the same verdict (test-retest);
stop early once every audience's answer shares are within a margin you set.
The sales rehearsal. Your pitch goes to every buyer. They answer with their objection, your rebuttals answer back, and they decide again. You get the objections ranked, which rebuttals worked, and a lead score per person ("who to call first").
The analyst panel. Five analysts read the whole run independently and each recommends go, iterate or no-go. Their agreement is a check on the report, not a vote that decides it.
5. Decide
The decision card puts go, iterate or no-go at the top, with the signals behind it: the survey's purchase intent (go needs a mean of 3.75 on the 1-5 scale, iterate 3.1), the feed test's best ad, the analysts, the top objection. The report explains it in plain language: who was simulated, what they said, what stopped them, what to change, an action plan and the limits. Every number in it is checked against the run's fact sheet; if the writing model fails, a data-only report is produced instead.
It all lands in one HTML dashboard that opens without a server and works offline, in light and dark, on a phone. Every chart has a table behind it, and the report downloads as Markdown, JSON, CSV or PDF. A live control room shows a run while it happens: each stage, every person lighting up as they act, and what they post. See the dashboard.
6. Check against reality
Simulated numbers mean little until they are compared with real ones. Import your real ad results or survey answers and MarketMirror:
backtests its predictions against your past ads;
calibrates the simulated survey against real answers: overlap, per-audience trust, and a corrected conversion rate;
forecasts clicks or sign-ups for a budget, using what it learned from those comparisons;
suggests the next tests from your real ad export and the run.
That loop, from simulation to a small real test and back, is the point: the simulation narrows down what to test; reality decides.
Use cases
Each recipe starts from a prepared simulation (./mm cli quickstart, then market-test with "approve_spend": true). Ask your agent in plain words, or run the command yourself. Results land in the run folder and in the dashboard section named in the last column.
Start with the decision you need to make:
Question | Test | Inspect |
Which ad should we try first? | Feed test with repeated seeds | Attention, clicks, reactions, and audience differences |
Does the offer feel trustworthy? | Survey and offer study | Intent, objections, and variation across seeds |
What price should we test? | Price sensitivity or tier comparison | Acceptable range and simulated choices |
Where does the website lose buyers? | Browser journey | Action trace, screenshots, and outcome checks |
How should sales answer objections? | Pitch and rebuttal rehearsal | Before/after responses and objection themes |
Does this match observed outcomes? | Holdout backtest and calibration | Errors, coverage, and validation limits |
You want to know | Ask your agent | Or run | Look at |
Which ad to run | "Test these 3 ads on our buyers in realistic feeds, 3 seeds." |
| Feed test, At a glance |
Which message or name lands | "Split-test these headlines; which one makes people want to try it?" |
| Messages, Results |
What to charge, and which offer | "Would people start a trial with a card required? What price feels fair?" |
| Pricing, Objections |
Whether a change really wins | "Compare no-card vs card-required vs a lower price on the same people, 3 seeds each." |
| Study, Scenarios |
Which country and audience to start with | "Where does this land best: India, the Philippines or Kenya?" | personas spec with segments and countries, then | Where they are (map), Audiences |
How a launch plays out socially | "Run our launch posts and day-3 announcement through a few social rounds." |
| Social feed, Activity |
How news changes the conversation | "A competitor just cut its price; push that into the running simulation." |
| Social feed (next round) |
Whether the email gets opened | "Test these subject lines and a 3-step sequence in their inboxes." |
| |
How a sales pitch is received | "Pitch every buyer, answer their objections, and tell me who to call first." |
| Sales rehearsal, lead scores |
Whether the website converts | "Send 3 buyers to our product page and see if they add to cart." |
| Website test report |
What real people already say | "Find what people say about home workout apps in India, with links." |
| Voices, Report |
What to test next, from real ads | "Read our ad export and suggest the next three tests." |
| Test ideas |
What results to expect | "Forecast sign-ups for USD 500 of spend, calibrated on our past campaigns." |
| Predictions |
Whether the simulation matches reality | "Compare the simulated survey with our real answers." |
| Evidence, calibration verdict |
Why someone said no | "Ask person 12 why they would not start the trial." |
| (answer printed) |
Who blocks the deal and who signs | "Run a buying committee on 12 households or companies." |
| Marketing lab |
How to make an ad better | "Rewrite this ad three times and re-test each version on the same people." |
| Marketing lab |
How to answer objections | "Build a battlecard from their real objections and test the answers." |
| Marketing lab |
Which tier people pick | "Basic, Plus or Family at these prices: what share picks each?" |
| Marketing lab |
What happens if a competitor moves | "Would they switch if X cuts its price by 30%?" |
| Competitors |
Whether the landing page holds attention | "Tear down our landing page with 20 buyers." |
| Landing and sales lab |
Whether different models agree | "Run the same market on Claude, Codex and Gemini people together." |
| Do the models agree? |
What someone thought but did not say | "Show me what people thought while they scrolled." |
| Agent thoughts |
Where a theme shows up in the run | "Find everything people said about price in Kenya." |
| (results printed) |
What is happening right now | "Show me the run live." |
| Live control room |
Simulated people remember what happened to them across tests in the same run: someone who hid your ad or unsubscribed from the email carries that into the survey and the pitch. Each study cell keeps its own memory, so scenarios never leak into each other. Set "memory_scope": "off" in a config to test every step on fresh people.
Feed-test previews cover Facebook, Instagram, Reddit, TikTok, LinkedIn, and X. Optional social rounds use the MiroFish/OASIS Twitter and Reddit environments. Search, organic, referral, and sales test families adapt the existing survey, inbox, and pitch engines; they do not reproduce those channels' production systems.
Example results: simulated market studies
These are outputs from documented local runs for Fitloop, an invented fitness app. They demonstrate the workflow and its limitations. They are not observed customer conversion rates or proof of predictive accuracy.
At a glance
Example | Scope | What it surfaced |
2,999-buyer mixed market | Claude-, Codex-, and Gemini-written buyers across six audiences | Trust objections and substantial differences between answering models |
150-buyer offer study | Three offers × three seeds on the same synthetic cohort | The card requirement reduced simulated willingness to start in every seed |
1,008-buyer launch test | Feed test, social rounds, survey, and report | A conflict between the ad's no-card promise and the landing page's card requirement |
Trial offer | Mean simulated “would start” | Min–max across 3 seeds |
No card · USD 4.99 | 4.1% | 2.7–5.4% |
Card required · USD 4.99 | 0.2% | 0.0–0.7% |
No card · USD 2.99 | 6.7% | 5.3–8.0% |
Seed ranges describe simulation variation. The next step is a small randomized real experiment; the price cut did not establish a predictive win.
The 3,000-person mixed market
One Fitloop market of 3,000 requested people, a third each written and interviewed by Claude Sonnet, Codex (gpt-6-luna) and Gemini 3.8 Flash (gemini-3.8-flash-low). Six audiences in India, the Philippines and Kenya, and five communities: Bengaluru Fitness Buddies (WhatsApp), Working Moms & Dads India (Facebook), Fit Pinoy Community (Facebook), Nairobi Runners & Gym Rats (WhatsApp) and r/budgetfitness; each person joins at most 2. The offer as advertised: a 7-day free trial with no card, then USD 4.99 a month. As in the 1,000-person run, the brief and the landing page still ask for a card at sign-up, and the test is meant to catch that.
Decision: iterate (medium confidence). The survey alone says no-go (purchase intent 2.88 of 5, 0.7% would start the trial, n = 2,999), the feed test says iterate (best ad fb_parents, expected CTR 1.02%), and 5 of 5 simulated analysts say iterate. Fix first: trust ("looks like a scam"), doubt that it is really free, and an unclear product.
What was measured | Result |
Would start the trial | 0.7% yes, 50.4% maybe, 48.9% no (2,999 of 2,999 answered) |
Purchase intent from free-text answers (1-5) | 2.88 ± 1.14; interest 3.22 |
Top objection | Trust / "looks like a scam": 59.6%; price 20.4%; unclear what it is 9.1% |
By audience, would start | Philippines young professionals 1.7%, Philippines parents 0.9%, India parents 0.9%, India young professionals 0.6%, students 0.3%, Kenya young professionals 0.0% |
Best survey message | "A month of Fitloop costs less than one gym visit" (about 1% yes, like the others) |
Acceptable price range | USD 1.50-5.50, sweet spot about USD 2.20 |
Test-retest (30 people, a simulated week later) | 76.7% gave the same verdict; 93.3% rated interest within 2 points |
Sales pitch (2,997 of 2,999 answered both turns) | Before the rebuttal 1.1% yes, 64.0% maybe, 34.8% no; after it 1.1% yes, 44.6% maybe, 54.2% no. The trust rebuttal moved 44.9% of those who raised trust, the price rebuttal 8.9% |
Social rounds (4 Reddit rounds) | 3,565 actions: 1,375 comment likes, 1,259 chose to do nothing, 770 comments, 135 post likes, 18 dislikes, 7 posts |
Analyst panel | 5 of 5 iterate (3 played by Claude, 2 by Codex). Themes: no local payment (UPI), a USD price against free YouTube, trust needs proof, a card wall kills the trial, data cost on cheap phones |
The feed test, five ads across 5,140 person-platform sessions:
Ad | Platform | Impressions | Thumb-stop | CTR | Expected CTR |
| 858 | 17.9% | 2.2% | 1.02% | |
| 749 | 21.5% | 2.3% | 1.02% | |
| 750 | 19.9% | 1.9% | 0.91% | |
| TikTok | 2,141 | 18.3% | 0.8% | 1.42% |
| 642 | 15.3% | 0.5% | 0.41% |
The pitch made things worse, not better: after hearing the rebuttals, about a fifth of the people moved from "maybe" to "no" (pitch score 23 of 100). The report's fix: rewrite the trust and price rebuttals to show the exact charge after day 7, the cancel steps and the local price instead of arguing, which matches the analysts' themes.
Responses differed across the writer/answerer cohorts. Split by the provider that wrote each person:
People written by | Played in rounds, survey and pitch by | Would start: yes / maybe / no | Top objection |
Claude (1,000) | Claude Sonnet 5.5 | 1.3% / 75.6% / 23.1% | trust, then price |
Gemini (1,000) | Claude Sonnet 5.5 (stand-in) | 0.9% / 69.7% / 29.4% | trust, then price |
Codex (999) | Codex | 0.0% / 5.7% / 94.3% | trust, then "unclear" |
The Gemini-written cohort, played by Claude, answered more like the Claude-written cohort than the Codex-written cohort. This suggests checking sensitivity to the answering model, but the cohorts contained different people and the writer/answerer combinations were not fully crossed. This run does not isolate a causal answering-model effect or prove it matters more than the writer. The rates describe these simulated cohorts; they are not forecasts of customer demand.
How the run went. The population took 2 h 57 min to prepare (with one restart), and the feed test 62 min through the three login gateways. The social rounds then ran into usage limits: four times Gemini's or Claude's login hit its limit during the rounds, and each hit started the four rounds over, usually after waiting hours for the reset. Then Gemini's login ran out of quota for about four days (Flash and Argon alike). So the rest of the run moved to an Anthropic API key: Claude Sonnet 5.5 played the Claude-written people and, as a stand-in (MM_PROVIDER_STANDIN=antigravity=claude), the Gemini-written ones; Codex kept playing its own people through its login. From there it ran without a pause: four rounds in 40 minutes, the survey of 2,999 with its retest in 23 minutes, the pitch to 2,999 people in 37 minutes, then the panel and the report. The API spend for that part is on the Anthropic console; MarketMirror did not meter the tokens inside the rounds, survey and pitch.
The population behind it:
Claude | Codex | Gemini | All | |
People written | 1,000 | 999 | 1,000 | 2,999 |
Interviewed | 1,000 | 987 | 1,000 | 2,987 |
With a decision card | 980 | 933 | 992 | 2,905 |
One requested person was not written because their calls failed. They are counted, not filled in.
The 12 Codex people whose interview failed are kept at depth
profile. 82 interviewed people have no decision card.Audiences: Students 642, Philippines young professionals 536, India young professionals 534, India parents 429, Philippines parents 429 and Kenya young professionals 429, each split evenly across the three providers. By country: India 1,284, the Philippines 1,125 and Kenya 590.
The stereotype guard changed 20 people and found no audience with too little variety.
With 7 actors from the knowledge graph and the 5 community accounts, the simulation has 3,011 agents, each with a profile.
The feed test made 1,289 model calls (Gemini 442, Codex 440, Claude 407) with 19.9M input and 0.94M output tokens, and lost no session.
Which offer should Fitloop launch with?
A study ran three offers on the same 150 simulated buyers, three seeds each (9 cells, about 10 minutes per cell through the local Claude Code gateway). Each number is the mean across seeds, with the lowest and highest seed in brackets; the brackets show simulation variation only.
Measure | No card, USD 4.99 | Card required, USD 4.99 | No card, USD 2.99 |
Purchase intent (1–5) | 3.51 (3.46–3.60) | 3.01 (2.99–3.04) | 3.60 (3.59–3.60) |
Would start the trial (%) | 4.1 (2.7–5.4) | 0.2 (0.0–0.7) | 6.7 (5.3–8.0) |
Would click (%) | 7.7 (3.3–16.5) | 10.9 (8.0–12.7) | 23.5 (21.3–25.3) |
Trust (0–10) | 3.6 (3.4–4.0) | 3.4 (3.3–3.4) | 4.1 (4.1–4.1) |
"Looks like a scam" as top objection (%) | 54.6 (47.5–59.7) | 88.2 (86.7–90.7) | 60.5 (58.0–62.7) |
Decision across seeds | iterate in 2 of 3 | no-go in 3 of 3 | iterate in 2 of 3 |
What the study concluded. Requiring a card was a clear loss in every seed: intent fell by 0.5, would-start fell by 3.9 points, "maybe" answers fell by 44.8 points, and scam worry rose by 33.6 points, all outside the seed spread. Dropping the price to USD 2.99 lifted would-click (+15.8 points) and trust (+0.47) beyond the spread, but its intent gain (+0.09) stayed inside it, so the price cut is not a clear winner even in simulation. It came out best in 2 of 3 seeds, and the study's summary marks its result as not supporting a predictive claim. The recommendation: keep the trial card-free, work on trust (proof, a named company, a clear "no card needed" line), and run a small randomized real test of USD 2.99 against USD 4.99. The full write-up is generated as study_report.md next to study.html in the study folder.
The 1,000-person run: what 1,008 simulated buyers said
Offer: 7-day free trial without a card, then USD 4.99 a month, shown to young professionals, parents and students in India, the Philippines and Kenya.
Measure | Result |
Answered the survey | 1,001 of 1,008 |
Would start the trial | 1.1% yes, 74.8% maybe, 24.1% no |
Purchase intent from free-text answers (1–5) | 3.07 ± 1.12 |
Top objection | "Looks like a scam / trust": 70.3% |
Most willing audience | Kenya young professionals: 2.8% yes, 8.3% would click |
Least willing audience | Students: 0% yes (their answers were too similar to trust; flagged in the report) |
By country, would start | Kenya 2.0%, India 0.9%, the Philippines 0.8% |
Best survey message | "A month of Fitloop costs less than one gym visit": 1.5% yes vs 0.9% for the others |
Acceptable price range | USD 1.80–7.99, sweet spot around USD 2.49 |
Test–retest agreement (30 people, a simulated week later) | 86.7% gave the same verdict |
Analyst panel | 5 of 5 analysts: iterate |
What it concluded: iterate (medium confidence). The signals differ: the survey alone points to no-go (n = 1,001), the feed test to iterate (best creative fb_parents, expected CTR 1.58%), and all five simulated analysts to iterate. The decision written at the end of the run, before the analyst panel was added, was no-go. The idea interests people (three in four said maybe), but trust blocks them. The report also caught a concrete flaw: the ad promises "no card", while the landing page form asks for a card number, expiry and CVV, and simulated people dropped off at that step. The next step it recommends is to remove the card step, add proof from real users in each country, and retest. These are simulated people; treat every rate as a direction until a real experiment confirms it.
Preparation, active simulation turns, surveys, email/feed reactions, pitches, and reports can all use model calls. Repeated seeds multiply work. Local gateways report upstream usage when available and label estimates; subscription use does not imply a measured monetary cost. Measure a small run before choosing a large cohort. First-time embedding setup may download model assets even when inference runs locally.
Measured runs
These are local Fitloop measurements, not speed guarantees. Model, hardware, activity settings, concurrency, and provider limits affect runtime. Every rate in this section is simulated.
Run | Simulated buyers | Scope | Measured wall time | Feed-test calls | Invoiced cost |
Fitloop launch | 28 | Marketing pipeline | 28m 12s | 75 | Unavailable |
Fitloop | 150 | Marketing pipeline | 54m 50s | 262 | Unavailable |
Fitloop mixed market check | 12 (4 each from Claude, Codex and Gemini) | Feed test, 2 social rounds (37 actions), survey, pitch, through the three login gateways | 17m 50s | 23 (Claude 8, Gemini 8, Codex 7) | Subscriptions (no invoice) |
Fitloop large (social only) | 1,008 | Six Reddit social rounds only | 3m 42s | Not measured | Unavailable |
Fitloop 1,000 | 1,008 | Full pipeline on Sonnet via the Claude Code gateway: feed test, 4 Reddit rounds, survey of 1,008 + retest of 30, 5 analysts, report | about 52m of active work | 434 | Subscription (no invoice) |
The 1,000-person run's active time breaks down as feed test 10m 17s (434 calls, 1,296,106 input and 375,768 output tokens), four social rounds 18m 28s (1,159 actions), survey and retest 20m 29s, report 1m 26s; the analyst panel failed during that run (all five analysts; fixed afterwards) and was run again later. Its wall-clock time was much longer: the Claude Code session limit paused it for 4h 12m mid-survey, it was resumed by hand the next morning, and the four social rounds were run again, so the active times above count only the final pass. API providers apply their own rate limits and quotas.
Preparing the people is measured separately, through the three login gateways. "Writing the people" is the persona planner alone (personas, interviews and follow-ups); "upload to ready" adds the knowledge graph, agent profiles and simulation config. At full depth the decision card comes from the follow-up call, so that column is also the number followed up.
Run | Depth | People (Claude / Codex / Gemini) | Interviewed | Decision card | Not written | Writing the people | Upload to ready |
Full-depth speed check | full | 60 (20 / 20 / 20) | 60 | 57 | 0 | 6m 7s | 7m 16s |
Fast population check | fast | 150 (45 / 55 / 50) | 24 | 150 | 0 | 7m 29s | 8m 51s |
Fitloop 3,000 mixed market | full | 2,999 (1,000 / 999 / 1,000) | 2,987 | 2,905 | 1 | 2h 40m 11s, with a restart | 2h 57m 31s, with a restart |
The 60- and 3,000-person runs used Sonnet, gpt-6-luna and gemini-3.8-flash-low. The 150-person run used Sonnet, Codex's default model and gemini-3.8-flash-medium; its 24 interviews are 4 per audience. It predates the even per-audience split (each audience was split 20 + 5 between two providers, hence 45 / 55 / 50); current runs split every audience evenly across all providers.
The 3,000-person times are not a clean single pass. At 14:34, after 1,865 of 3,000 people, the backend was restarted to apply a lane fix (hung calls had cut the Gemini lane from 10 to 2; now only a rate limit halves a lane). The new prepare task replayed 1,043 finished answers from the population cache and wrote the rest; that second part alone took 1h 54m 39s. After the people were written, the 3,011 agent profiles took about 7.5 minutes and the simulation config about 9.5 minutes.
Quick start
Requires Git, Bash, Python 3, and uv. The installer provisions the Python 3.12 environment. Use Linux, macOS, or WSL; native Windows has not been validated.
git clone https://github.com/DarkWorldCoder/market-mirror.git
cd market-mirror
bash install.sh
./mm configureChoose your model provider, then start the backend and create your first scenario:
./mm provider check
./mm start
./mm cli quickstart --out scenarios/acmeThe wizard asks about your product, buyers, channels, budget, and test size. It writes editable inputs, prepares the simulated people, and stops for review before running the remaining tests. Preparation itself uses model calls. Start with a small cohort. provider check discovers models without running inference; it does not test model quality or tool support.
Inspect the generated files and profiles. To continue, retain the returned simulation_id, set "approve_spend": true in scenarios/acme/market_test.json, and run:
./mm cli market-test scenarios/acme/market_test.json
./mm live --run-dir mirofish_runs/acmeThe live viewer runs at 127.0.0.1:5050, and its control room at 127.0.0.1:5050/live shows every stage, every simulated person lighting up as they act, what they post, and the progress log, updated every two seconds. Use the actual run directory printed by the pipeline if its name differs. The saved dashboard.html also opens directly without a server. To generate templates without preparing people, use ./mm cli quickstart --no-run.
If a market-test run stops (a crash, a restart, a closed terminal), first make sure the earlier market-test process has exited and ./mm cli status --simulation <sid> shows no rounds running (wait for them, or stop them with ./mm cli stop), then run the same command again, with the same MIROFISH_URL for a backend on another port. It resumes from the run folder's run_manifest.json, and a simulation that stopped before its people were ready is prepared again; people already written come from the cache. With "resume_steps": true in the config it also reuses a usable saved feed test and survey and a live social environment. The wait for preparation grows with the market (at least an hour, or about 6 s per person), and backend tasks are followed for up to six hours.
./mm start starts the backend; ./mm cli start --simulation … starts social simulation rounds. All examples below use the repository's ./mm helper—no global command or alias is required.
Test your product URL
Install the optional browser support, then use a prepared simulation's ID:
uv sync --project backend --extra browser --python 3.12
backend/.venv/bin/python -m playwright install chromium
./mm cli website-test --simulation "$SIMULATION_ID" \
--url https://your-product.example/products/shoes --goal add_to_cart \
--task "Evaluate these shoes and decide whether to add a suitable pair to your cart." \
--limit 3 --steps 12Restart the backend after installing the browser extra. Replace the URL with your site. Browser control supports search and keyboard actions, product options, hover menus, same-site tabs and frames, open shadow DOM, and bounded recovery when the page changes. Add --record-video for a recording or --headed to watch on a desktop with a graphical session. Agents may interact with the actual cart; signup submission requires explicitly supplied test data and opt-in. No orders or payments are supported. Named DOM checks verify outcomes independently of what an agent claims. Without them, completion stays unverified. See real website testing for outcome checks, signup setup, browser limits, and the evidence report.
Browser runs use compact model observations while retaining full local evidence. Set call, input-token, and elapsed-time limits; repeated actions on unchanged pages are detected separately from buyer departures. Reports show available cache usage, model time, and cost basis. See browser context and run limits.
Choose your model
Provider | Setup | Authorization |
Local Codex |
| Existing local Codex login |
Local Claude Code |
| Existing local Claude Code login |
ChatGPT sign-in |
| Eligible account; scope and model access checked during setup |
Local Antigravity (Gemini models) |
| Existing local Antigravity login; |
Local Gemini CLI |
| Existing local Gemini CLI login |
Every installed login behind one gateway |
| The login CLIs that are installed and signed in ( |
Ollama / LM Studio | Choose the provider in | A locally installed, tool-capable model |
Claude / OpenAI / Google Gemini API | Choose the provider in | Provider API key ( |
Other API providers: DeepSeek, Kimi (Moonshot AI), Mistral, xAI (Grok), Groq, Together AI, Fireworks AI, OpenRouter, Qwen (Alibaba Model Studio), Z.ai (GLM), Cerebras, MiniMax |
| That provider's API key ( |
Other compatible servers | Configure the endpoint and model, or the pool slot | Provider-dependent |
A login gateway uses the subscription you already have, with its usage limits. API providers bill usage separately and apply their own rate limits and quotas. Model choice, gateway settings, usage-limit pauses, prompt caching with a Claude API key, the settings page and API costs for 3,000 people are in providers, mixed markets and large runs.
The default knowledge graph uses local SQLite and local embeddings; no Zep account is required. Model downloads and public-source research require network access. Hosted providers and CLI gateways receive model prompts through the selected provider. Model usage remains subject to that account's limits and charges.
Several models in one market
One market can be played by several providers at once, for example Claude, Codex and Gemini. Each provider writes an equal share of every audience, interviews its own people, plays them in the feeds and social rounds, and answers for them in the survey and pitch, so one model speaks for a person from start to finish. The dashboard then asks "Do the models agree?": it splits the results by provider and checks whether their would-convert ranges overlap and whether they name the same top objection and best message. Agreement is a robustness check on the simulation, not evidence about real buyers.
LLM_POOL=claude_code:sonnet,codex:gpt-6-luna,antigravity:gemini-3.8-flash-low
LLM_POOL_STRICT=true # rounds and tests stay on each person's own providerEach provider gets its own lane of calls in flight, so a slow one never holds up a fast one, and a provider whose account runs out can move to an API key or be stood in for by another provider mid-run. The full setup (gateways on separate ports, tokens, lanes, depth settings for thousands of people) is in providers, mixed markets and large runs.
Connect your agent
Use MarketMirror from Codex, Claude Code, Cursor, Gemini, or another client supporting its MCP transport. Install and configure the engine first using the quick start above.
Codex plugin
From the repository root:
codex plugin marketplace add "$PWD"
codex plugin add market-mirror@market-mirror-localClaude Code plugin
claude plugin marketplace add "$PWD"
claude plugin install market-mirror@market-mirrorThe plugin bundles the main workflow plus focused research, planning, testing, and evidence-review Agent Skills. They share one canonical source. Client plugin support and configuration locations vary; see the plugin guide.
Gemini CLI extension
gemini extensions install https://github.com/DarkWorldCoder/market-mirrorThe extension adds the MarketMirror MCP server and a short GEMINI.md guide. Install the engine with the quick start above, or ask Gemini to run setup.
Any compatible MCP client
The quickest way is the PyPI package (needs uv); it is also listed in the official MCP Registry as io.github.DarkWorldCoder/market-mirror:
{ "mcpServers": { "market-mirror": { "command": "uvx", "args": ["marketmirror"] } } }Then ask your agent to run doctor, and setup if the engine is not installed yet. To use a checkout you already have instead, print configuration for your client without overwriting its settings:
./mm agent-config --client generic
# Also supports: codex, claude, cursor, geminiThe output contains an absolute server path and engine location. Copy the entry into your client's MCP configuration and restart the client. The stdio server can also be launched with ./mm mcp.
For authenticated loopback HTTP MCP:
python3 plugin/skills/market-mirror/scripts/mcp_server.py \
--http --port 5061 --token-file /absolute/path/to/private-tokenCreate a private token file containing a random bearer token of at least 24 characters. Connect to http://127.0.0.1:5061/mcp using that token. This transport is intended for local agents, not a hosted remote service.
Try this in your agent:
Use MarketMirror to analyze our product and campaign performance. Bring in ad and analytics data from tools you have authorized access to, or use the exports I provide. Research cited customer feedback, survey a small simulated audience, and compare two ads or offers. Test our product URL with synthetic visitors, show the evidence and missing inputs, and recommend the next real experiment.
Core tools: doctor, provider_setup, understand, ads_import, ideas, website_test, research, actuals_import, campaign_plan, run_test, study, job_status, holdout_backtest, predict, and results. Long jobs return status, logs, and result paths. Poll job_status before treating a job as complete.
Example workflow
flowchart LR
A[Product brief] --> B[Product profile]
C[Cited public feedback] --> D[Campaign plan]
J[Connected ad data or exports] --> D
B --> D
D --> E[Surveys, campaign tests and website journeys]
E --> F[Compare scenarios]
F --> G[Next real experiment]
G --> H[Import observed results]
H --> I[Backtest and calibration]
I --> DAfter quickstart creates scenarios/acme/, replace its example content with your product material:
./mm cli understand --docs scenarios/acme/brief.md --name Acme \
--out scenarios/acme/product_profile.json
./mm cli research --query "home workout app" --country in \
--out scenarios/acme/research
./mm cli campaign-plan --profile scenarios/acme/product_profile.json \
--research scenarios/acme/research/voices.json \
--goal "trial signup" --budget 500 --currency USD \
--channels search organic paid_social email sales referral \
--out scenarios/acme/campaign_plan.jsonThe plan contains test_specs, cited evidence, assumptions, and unknowns. Channel budgets start as equal draft allocations. Review the claims and allocations, save a selected spec as experiment.json, and test it on a prepared simulation:
# Set this to the ID returned by your preparation step.
SIMULATION_ID=sim_replace_me
./mm cli test --simulation "$SIMULATION_ID" --spec @experiment.json
./mm cli study --spec study.jsonexperiment.json and study.json are your reviewed input files; input examples and ./mm cli study --help describe their contracts. Study cells reuse completed work; rerun the same specification to resume. Each independent comparison should begin from the same initial state and matched scenario seeds.
Import actual outcomes when available. This example expects an export with Day, Country, Views, and Clicks columns:
./mm cli actuals-import exports/campaign.csv --source-type ads \
--map date=Day market=Country impressions=Views clicks=Clicks \
--out scenarios/acme/actuals.json
./mm cli holdout-backtest --pairs experiments_with_train_and_holdout.json \
--out scenarios/acme/holdout_backtest.jsonThe importer retains source, date, market, and quality labels and returns row-level validation errors. Backtests require explicitly paired simulated and real outcomes with declared training/holdout splits; the imported export alone is not a ready-made holdout dataset.
./mm cli profiles --simulation "$SIMULATION_ID" --brief
./mm cli feedtest --simulation "$SIMULATION_ID" --creatives @creatives.json --seeds 3
./mm cli survey --simulation "$SIMULATION_ID" --offer "Acme trial" \
--goal "start a trial" --messages @messages.txt --split --pricing USD
./mm cli emailtest --simulation "$SIMULATION_ID" --emails @emails.json
./mm cli pitch --simulation "$SIMULATION_ID" --script @pitch.md --rebuttals @rebuttals.md
./mm cli report --simulation "$SIMULATION_ID" --run-dir mirofish_runs/acmeIf preparation saved a complete profile set but stopped before its configuration was ready, use ./mm cli prepare --simulation "$SIMULATION_ID" --resume-only. It reuses saved profiles and refuses to regenerate incomplete sets.
Keep inputs under scenarios/ and outputs under mirofish_runs/; both are gitignored. Use ./mm cli --help for the full command list.
What it adds to OASIS and MiroFish
MarketMirror is built on MiroFish, which runs on OASIS. OASIS gives a realistic social platform (feeds, posts, follows and a recommendation system) where language-model agents act. MiroFish adds a knowledge graph and a prediction report on top. MarketMirror turns them into a market test that ends in a decision:
MarketMirror adds audience planning, marketing experiments, and decision reports to that foundation:
Layer | Role in the workflow |
Social simulation environments with agent actions and recommendation feeds. | |
The upstream application foundation for knowledge graphs and simulation reports. | |
MarketMirror | Segmented synthetic buyers, ad and offer tests, surveys, sales rehearsals, provider comparisons, offline reports, and calibration workflows. |
Evidence and simulation limits
Generated interviews remain synthetic. Detailed biographies and BDE profiles do not turn invented people into customer evidence.
Behavior controls are assumptions. Attention, fatigue, social proof, and landing friction include hand-set priors that need validation for the intended market.
Seed variation measures simulation variation. It is not a predictive confidence interval. Repeated agreement and synthetic test–retest consistency do not establish agreement with humans.
Backtests answer specific questions. The existing no-persona comparator is the mean real outcome in the training set. Beating it alone does not establish general predictive reliability or replace a matched generic-reader LLM comparison.
Calibration depends on coverage and assumptions. Uncalibrated predictions omit intervals and success probabilities. Validate absolute rates, rankings, and subgroup findings separately on untouched real experiments.
Read the human-behavior research guide for the papers, inspected code gaps, and proposed memory/state architecture. It distinguishes implemented controls from future work and experimentally supported results.
Frequently asked questions
Are these real buyers or people I can contact?
They are AI-generated characters inside the simulation. Their names, stories, responses, and lead scores describe those characters. They are not a contact list of real prospects.
Will MarketMirror run my ads or send emails?
It simulates ads, inbox reactions, and sales conversations. It does not publish ads, send outreach, or change your ad account.
Can buyers use my actual website?
Yes, with optional browser support. Synthetic visitors can inspect real pages and attempt tasks such as finding a product or adding it to a cart. Reports include screenshots and action records. Explicit page checks are needed to verify task completion. Orders and payments are not supported; see website testing for supported actions and signup requirements.
Do I need to write code?
You can request tests in plain language through a connected AI assistant. Installing MarketMirror, choosing a model, and connecting that assistant require technical setup. The command-line tools are also available for direct use.
Is it free?
MarketMirror's source code is available under AGPL-3.0. Running it may consume paid AI usage, subscription allowances, or local computing resources. Audience size, interview depth, tests, and repeats affect the work required. Measured runs give examples, not cost or speed guarantees.
Can I bring real research and campaign data?
Yes. You can supply product documents, linked feedback, and campaign exports. MarketMirror can also collect supported public feedback with source links. Comparing simulations with actual outcomes requires suitable paired data; importing an ad export alone does not create a complete validation study.
What do the technical terms mean?
Term | Plain-language meaning |
Persona / synthetic buyer | An AI-generated character representing a type of buyer. |
Audience segment | A group you want to explore, such as students or working parents. |
Creative | The image, video idea, or copy used in an ad. |
Split test / A/B test | Showing different versions to different people to compare responses. Here those people are simulated. |
CTR | Click-through rate: the share of ad views that produced a click. In a simulated test, both are generated events. |
Seed | A random starting setting used to repeat a simulation under different draws. |
Backtest | Comparing a simulation with known outcomes from past experiments. |
Calibration | Adjusting the simulation's estimates using observed results. It still needs independent evaluation. |
Other integrations
Interface | Entry point |
REST API | Backend at |
Python SDK |
|
Docker | Configure |
Webhooks | Set |
The Docker services mount uploads, scenarios, and run outputs from the host. The host's loopback login gateway is not directly reachable from those containers; use a provider reachable from the container or run the CLI gateway workflow on the host.
Development
cd backend
uv sync --python 3.12
uv run --no-sync python -m pytest tests claude_cli/tests -qTests mock model calls and do not need a live model key. CI also checks generated API documentation. Backend services live in backend/app/, the CLI and report renderer in backend/claude_cli/, agent tools in plugin/, and the Python SDK in sdk/.
Documentation
You want to… | Read this |
Install MarketMirror and connect an assistant | |
Keep a separate copy of the detailed setup and reference | |
Choose AI providers or run a larger audience | |
Prepare buyer specifications, ads, and test files | |
Test an actual product website | |
Install the agent plugin | |
Build an integration | |
Understand the research and proposed improvements | |
Contribute or report a problem |
Support the project
If MarketMirror was useful, ⭐ star the repo so other people find it, and tell us what you tested. Every issue gets an answer.
Credits and license
MarketMirror builds on MiroFish and OASIS, adding audience planning, marketing tests, and decision reports to their simulation foundation. It is released under AGPL-3.0.
The dashboard embeds Apache ECharts; see third-party notices. The original MiroFish README is preserved. The banner and workflow illustrations were AI-generated; their generation prompts are included.
Available Tools
27 toolsactuals_importC
Normalize a local ads, email, website, CRM or survey CSV/JSON export, retaining provenance and row-level validation errors.
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| date | No | ||
| file | Yes | ||
| market | No | ||
| source | No | ||
| mapping | No | ||
| quality | No | ||
| source_type | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does disclose two meaningful traits: provenance is retained and row-level validation errors are preserved rather than aborting the import. It says nothing about whether files are written to disk (the 'out' param hints at it), overwrite behavior, idempotency, or permission requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler; every clause adds information (source types, provenance, error retention). It is efficient, though the terseness borders on under-specification for a tool with eight parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 8-parameter tool with a nested object, zero schema coverage, no annotations, and no output schema, the description leaves too much unknown: what 'mapping' and 'quality' control, what 'out' does, and how validation errors are surfaced to the caller. The complexity clearly outstrips the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 8 parameters, including a nested 'mapping' object and ambiguous fields like 'quality', 'source', and 'date'. Only 'file' and 'source_type' are inferable from the prose/enum; the remaining six parameters are undefined anywhere, and the description does not compensate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb ('Normalize') plus a concrete resource class (local ads/email/website/CRM/survey CSV/JSON exports) makes the operation distinguishable from generic import tools. However, it never addresses the obvious overlap with the sibling tool 'ads_import', leaving ambiguity about which one handles ads data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The enumerated source types imply the domain of use, but there is no explicit when-to-use, no exclusion, and no pointer to alternatives. With a sibling named 'ads_import' sitting in the same list, the absence of routing guidance is a real gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ads_importB
Turn a Meta/Google Ads export (CSV or JSON, absolute path) into a seed Markdown file plus benchmarks JSON.
| Name | Required | Description | Default |
|---|---|---|---|
| out | Yes | absolute path of the seed .md to write | |
| file | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden; it does disclose that the tool writes two artifacts (seed .md and benchmarks JSON) and accepts CSV or JSON input. It does not say whether it overwrites existing files, what happens to malformed exports, or what errors look like — meaningful gaps for a file-writing tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler that names input, input formats, and both outputs. It is perhaps too terse given the tool's file-writing side effects, but nothing in it is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations and no output schema, the description covers the essential input formats and the two produced artifacts, which is the minimum an agent needs to invoke it. It omits overwrite behavior, expected export schema, and failure modes, leaving gaps for a mutation-style tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: `out` is documented in the schema, but `file` has no schema description, and the description compensates by explaining it is a Meta/Google Ads export in CSV or JSON with an absolute path. That is real added meaning, but the format/validation expectations (e.g., required columns) remain unspecified.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (turn/convert) and both the input resource (Meta/Google Ads export) and outputs (seed Markdown + benchmarks JSON), so the agent knows exactly what the tool produces. It doesn't contrast with the nearby actuals_import sibling, which is the main thing a reader would want clarified.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this versus alternatives such as actuals_import or campaign_plan, nor any prerequisites or exclusions. The agent must infer the usage context purely from the tool name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
backtestB
Score a run's feed test against real ad results (from ads_import): rank correlation, winner match, calibration.
| Name | Required | Description | Default |
|---|---|---|---|
| map | No | ||
| run | Yes | ||
| actuals | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden but does not say whether this is a read-only computation or whether it persists results, nor whether it requires a completed run or imported actuals to exist. It discloses only the output dimensions (correlation, winner match, calibration) and nothing about side effects, errors, or cost.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single dense sentence with the action front-loaded and the metric list appended; no filler. It stops short of five only because the compactness comes at the cost of the missing prerequisite and parameter detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter, annotation-free tool with no output schema, the description does useful work by naming the returned metrics. But it omits the purpose of 'map', the prerequisite that ads_import/actuals_import must have run, and whether a run must be finished — gaps an agent would need to resolve by trial.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 3 parameters. The description implicitly covers 'run' and 'actuals' (real ad results from ads_import) but says nothing at all about the 'map' parameter, which is entirely undocumented in both schema and prose. Partial compensation is not enough at this coverage level.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Score') and resource ('a run's feed test against real ad results'), and enumerates the metrics produced (rank correlation, winner match, calibration), so the agent knows exactly what the tool computes. It does not explicitly distinguish itself from the sibling holdout_backtest, which is a similar scoring tool, leaving that differentiation to the agent.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The parenthetical '(from ads_import)' hints at the prerequisite source for actuals, which implies a call ordering. However, there is no explicit statement of when to use this versus holdout_backtest, and no exclusions or preconditions beyond that hint.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
calibrateA
Check a run's simulated survey against real answers (absolute path to a CSV with segment,country,gender,age,y where y is yes/maybe/no or 0-1; 50-300 rows is enough): covariate overlap, per-audience calibration, a trust / correct / walk-away verdict and an AIPW-corrected rate. Writes calibration.json into the run folder.
| Name | Required | Description | Default |
|---|---|---|---|
| by | No | ||
| map | No | ||
| run | Yes | ||
| field | No | ||
| real_csv | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description does real work: it discloses the side effect that calibration.json is written into the run folder, and names the verdict categories (trust / correct / walk-away) and the AIPW correction. It omits failure behavior (missing run, malformed CSV) and permission/overwrite semantics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One dense, front-loaded sentence that leads with the core verb and ends with the write side effect. The long parenthetical on CSV format is justified, though the sentence is heavy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No annotations and no output schema, so the description carries full load; it covers the deliverable file and headline metrics but leaves three parameters and error conditions undefined. Adequate to start, not complete for a 5-parameter tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and three of five parameters (by, map, field) are never mentioned, so an agent cannot know what they control or their formats. Only real_csv is explained in useful detail (columns, label encoding, row count) and 'run' is implied by 'run folder'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Check a run's simulated survey against real answers', and enumerates the concrete outputs (covariate overlap, per-audience calibration, verdict, AIPW-corrected rate). This clearly separates it from siblings like results or actuals_import.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies the tool is used after a run exists and that you supply an external ground-truth CSV, and even gives sizing guidance ('50-300 rows is enough'). But it never says when to prefer this over siblings such as results, backtest, or actuals_import, nor any prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
campaign_planB
Save an evidence-led campaign plan and runnable test specs from a profile or existing project. No campaign is sent.
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| goal | Yes | ||
| budget | Yes | ||
| profile | No | ||
| project | No | ||
| channels | No | ||
| research | No | ||
| profile_file | No | ||
| research_file | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It does disclose one important trait — no campaign is actually sent — but omits whether the saved plan persists where, whether it overwrites, permission needs, or what is returned. For a write-style tool with nine parameters, the disclosure is thin but not misleading.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the purpose front-loaded and the safety disclaimer second. Nothing is wasted, though the compactness comes at the cost of the parameter detail a tool this complex needs.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
A nine-parameter tool with nested objects, no annotations, and no output schema needs substantially more description. Key elements — budget shape, profile vs profile_file vs project precedence, channels, out, and return behavior — are all unexplained.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across nine parameters, including nested objects (budget, profile, research) and required goal/budget. The description only loosely gestures at profile, profile_file, project, and research via 'from a profile or existing project'; it does not explain the structure of budget, channels, out, or the file-vs-object alternatives.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('Save an evidence-led campaign plan and runnable test specs') and identifies the input modes ('from a profile or existing project'). It is clear enough to distinguish from siblings like research or pitch, though the term 'evidence-led' is loosely defined.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'No campaign is sent' usefully scopes what this tool does not do, and 'from a profile or existing project' hints at two entry paths. However, it never states when to choose campaign_plan over siblings such as pitch, research, or study, leaving selection to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cliB
Advanced: run any MarketMirror CLI command as a background job, e.g. ['survey','--simulation','sim_x','--offer','...']. See CLAUDE.md in the engine.
| Name | Required | Description | Default |
|---|---|---|---|
| args | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, yet it only discloses that execution happens asynchronously as a background job. It says nothing about whether arbitrary commands can mutate or delete data, what permissions or environment are used, how failures surface, or that results must be retrieved via job_status/results.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact sentences, with the 'Advanced' qualifier and the verb front-loaded before the example. The trailing pointer to CLAUDE.md is a cheap external reference an agent cannot read at call time, but it costs little space.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a catch-all command executor with no annotations, no output schema and an undocumented single parameter, the description is thin: it omits the return value (job handle), how to poll progress, timeout/rate-limit behavior, and any safety constraints on what commands are permitted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it partly does: the example shows args is an ordered argv list where element 0 is the subcommand and the rest are flags/values. It still gives no escaping, quoting, or shell-vs-direct-execution semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: run any MarketMirror CLI command as a background job, with a concrete example argv array. It is clearly distinguishable as a generic escape hatch rather than a targeted operation, though it never names which siblings it supersedes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'Advanced:' prefix implies this is a fallback when no dedicated tool fits, but that is inference, not guidance. There is no statement of when to prefer run_test, backtest, survey-workflow tools, or similar over raw CLI invocation, and no prerequisites or warnings about misuse.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
doctorB
Check MarketMirror: engine installed, Python deps, API key configured (never shown), backend running.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden; "Check" implies a non-mutating probe and the parenthetical "(never shown)" usefully discloses that the API key is redacted from output. However, it never states whether the tool has side effects, whether it can repair what it finds, or what form the report takes, which are exactly the gaps annotations would normally fill.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with a compact enumeration of the four checks; no filler. Slightly clipped to the point of ambiguity, but every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema diagnostic tool, the agent still needs to know whether this is a safe read-only probe and how to act on the results, neither of which is covered. Adequate to identify the tool, insufficient to complete the troubleshooting loop.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4; there is nothing parameter-level for the description to clarify or omit.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb ("Check") and enumerates the exact subsystems inspected: engine install, Python deps, API key, backend status. That is far more informative than the bare name "doctor", though it never distinguishes itself from look-alike siblings such as setup, provider_setup, or start_backend.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It implies a diagnostic/troubleshooting role but never states when to run it (e.g., before start_backend, after install, when a run fails) or when to prefer setup/quickstart instead. No exclusions or alternatives are named, so the agent must infer the workflow placement.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
email_testA
Email campaign test on the simulated people's inboxes: pass emails [{id, from_name, subject, preheader, body, cta, send_time (morning|midday|evening|night), segments?, sequence?: [{delay_days, subject, body}]}] plus optional list_size and benchmarks {cost_per_send, lead_value, close_rate, deal_value}. Each person gets a realistic inbox for the day; an attention gate decides opens, Claude decides skim/read/click/reply/unsubscribe/spam with honest odds. Returns open, click, click-to-open, unsubscribe and spam rates per variant and audience, subject-line ranking, sequence funnel, projection from list size to leads and revenue, spam-word warnings. Needs a prepared simulation (simulation_id). Background job.
| Name | Required | Description | Default |
|---|---|---|---|
| run | No | ||
| emails | Yes | ||
| list_size | No | ||
| benchmarks | No | ||
| simulation_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it explains the attention-gate simulation model, that an LLM decides skim/read/click/reply/unsubscribe/spam 'with honest odds', the async background-job nature, and exactly what metrics are returned. It omits cost/rate-limit implications but covers behavior far beyond a bare verb.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded, but the body is a single dense run-on packed with inline field lists, making it hard to scan. The information is useful, but structure (separating inputs from outputs) would let the same content earn its place better.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with nested objects, no output schema, and zero schema description coverage, the description supplies both input shape and expected return metrics plus the background-job note. Remaining gaps are the undocumented `run` parameter and no pointer to job_status for retrieving results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it largely does: the emails array is enumerated field-by-field including the send_time enum values and nested sequence structure, plus benchmarks keys and list_size. The exception is the `run` parameter, which is never explained anywhere.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource: an email campaign test run against simulated people's inboxes, with a defined outcome (open/click/unsubscribe/spam rates). This clearly separates it from siblings like website_test or market_test without needing to name them, though it never explicitly names those alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states the key prerequisite ('Needs a prepared simulation (simulation_id)') and that the operation is a background job, which implies polling job_status. However, it never says when to pick this over run_test/market_test or what to do if no simulation exists, leaving usage largely implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
holdout_backtestC
Evaluate independent held-out real experiments against a training-mean no-persona baseline.
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| pairs | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral-disclosure burden. It does not state whether the operation is read-only, what permissions are needed, whether it has side effects, or what the response contains.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single front-loaded sentence with no wasted words. It efficiently communicates the evaluation target and baseline.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with two parameters, no annotations, no output schema, and 0% schema description coverage, the one-sentence description is too sparse. It leaves the input format, output behavior, and operational context undocumented.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description never mentions the required 'pairs' parameter or the optional 'out' parameter. Neither parameter's meaning, format, or expected structure is explained anywhere.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Evaluate') and resource ('independent held-out real experiments') plus a comparison baseline. It is clear what the tool does, but it does not explicitly distinguish itself from the sibling 'backtest' tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit guidance on when to use this tool versus alternatives such as 'backtest' or 'run_test'. The phrase 'held-out' implies a usage context, but no when/when-not conditions or prerequisites are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ideasA
Next test ideas from real data: pass ad rows you fetched from a connected ads tool (Meta/Google/TikTok: name, campaign, spend, impressions, clicks, conversions, country...) and/or an export file path, plus a previous run (its objections, audiences and decision). Returns prioritised experiments, each citing its evidence, with the MarketMirror config to simulate it first. Writes ideas.json/.md into the run folder and adds a 'Test ideas' section to the dashboard.
| Name | Required | Description | Default |
|---|---|---|---|
| n | No | ||
| run | No | ||
| rows | No | ||
| brand | No | ||
| brief | No | ||
| ads_file | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description does real work: it discloses side effects (writes ideas.json/.md into the run folder and appends a 'Test ideas' section to the dashboard) and the shape of the return (prioritised experiments each citing evidence plus a simulation config). It omits permission/auth needs, what happens when no run folder exists, and whether writes are additive or overwrite.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The input contract is front-loaded before the outputs and persistence behavior, and both sentences carry real information with no filler. The opening clause is long and comma-heavy, which slightly slows parsing but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a six-parameter tool with zero schema coverage, no annotations, and no output schema, the description covers the data inputs, the run context, and the returned artifacts adequately. It still leaves three parameters undefined and gives no guidance on defaults or failure modes, which is a meaningful gap given the empty schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate for six bare parameters. It does well for rows (enumerating name, campaign, spend, impressions, clicks, conversions, country), ads_file (export file path), and run (prior run's objections, audiences, decision), but n, brand, and brief receive no explanation at all, leaving half the surface undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific action (generate next test ideas) on a specific resource (real ad data / a previous run) and names what comes out (prioritised experiments with evidence plus a MarketMirror config to simulate). It implicitly separates itself from market_test/run_test by positioning ideas as the pre-simulation step, though the verb phrasing 'Next test ideas' is slightly awkward.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when it applies ('pass ad rows you fetched... and/or an export file path, plus a previous run'), which tells the agent the expected preconditions. It never states when NOT to use it or names an explicit alternative sibling, so the agent must infer the boundary with market_test, research, or campaign_plan.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
job_statusC
Status, log tail and final result of a background job.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses that log tail and final result are returned, but says nothing about polling behavior, error handling for unknown jobs, permissions, or whether the call blocks or returns partial data.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with zero waste. It is appropriately sized for a simple one-parameter status tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, and the description usefully names the return components despite there being no output schema. However, it leaves the required job_id undocumented and omits usage context, so it meets only the minimum viable bar.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and the description never mentions the required job_id parameter. The parameter name is self-explanatory, but no format, source, or usage details are added beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the specific resource (a background job) and the specific data returned: status, log tail, and final result. That is clearer than a generic status tool, though it does not explicitly differentiate itself from siblings such as results or list_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool, when to poll versus use results/list_runs, or what prerequisites exist. The single vague context is that it applies to background jobs, leaving the agent to infer usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_runsC
List previous runs with their decision.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden but says nothing about read-only behaviour, ordering, result limits, pagination, or what a 'decision' looks like. Only the verb 'list' faintly implies a non-mutating read.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler, but it is terse to the point of under-specification rather than true conciseness. Nothing is wasted, yet little is conveyed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, no annotations, and no parameter docs, so the description is the only source of information about the return value and it does not describe it. An agent cannot know the result shape, ordering, or what 'decision' means before calling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate; baseline 4 applies. The one missed opportunity is stating that no filtering is possible, which would matter given the unbounded 'previous runs' scope.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a verb (list) and resource (runs) plus an extra field it returns (decision), which is more informative than the title alone. But 'runs' is undefined in a sibling set that also contains run_test, backtest, results and job_status, so an agent cannot confidently tell what a 'run' is or which tool surfaces them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No indication of when to call this instead of results, job_status or backtest, which are the obvious neighbours. The agent is left to guess whether this is a history view, a status poll, or a results summary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
market_testA
All-in-one market test. Config keys: name, title, seeds (absolute paths to documents), requirement (text), personas (persona spec object or absolute path) or auto_personas (int), creatives (feed-test object with creatives, landing, benchmarks), seeds_n (int, repeat the feed test with N seeds and report ranges), rounds (int), survey {offer, goal, messages (list), split, pricing}, panel {analysts, brief}, report (bool), simulation_id (reuse), approve_spend (bool). Without approve_spend it stops after creating the personas so you can show them to the user; then call again with simulation_id and approve_spend true. Background job.
| Name | Required | Description | Default |
|---|---|---|---|
| config | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does disclose the approval gate, persona-creation pause, simulation reuse, and background execution. It omits spend/cost implications, permissions, idempotency, error behavior, and how results are retrieved.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose and approval workflow are front-loaded, and most content is necessary given the empty config schema. However, the long run-on config key list is dense and hard to scan, reducing practical clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex asynchronous job with no annotations, no output schema, and an empty config schema, the description provides useful config semantics and the approval sequence. It still omits required/default config keys, result retrieval via job_status/results, and error or cost behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage and exposes only an undifferentiated config object, so the description's enumeration of keys and their meanings compensates substantially. Some nested structures are only loosely described, but the key semantics are much richer than the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening phrase 'All-in-one market test' plus the config-key list makes clear that this tool runs persona generation, feed creative testing, surveys, panels, and reporting. However, it does not explicitly distinguish itself from siblings such as run_test, website_test, or email_test.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a concrete two-step approval flow: call without approve_spend to stop after persona creation, then call again with simulation_id and approve_spend true. It also notes that the work runs as a background job. It lacks guidance on when to choose this over sibling test tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
pitchA
Sales rehearsal: pitch a script to every simulated buyer of a simulation (its environment must be alive: run market_test with rounds first, or start it), collect objections and the questions they ask back, then answer with rebuttals ({objection_type: text, '*': generic}). Scores the pitch, each rebuttal, and every person (lead score). Background job.
| Name | Required | Description | Default |
|---|---|---|---|
| run | No | ||
| script | Yes | ||
| channel | No | ||
| product | No | ||
| segment | No | ||
| rebuttals | No | ||
| simulation_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description still discloses meaningful traits: it is a background (async) job, it mutates state by scoring the pitch and assigning lead scores to every person, and it requires a live environment. It does not explain how to retrieve the async result (job_status/results) or what happens if the simulation is not alive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with 'Sales rehearsal' and the core action, and every clause adds information (precondition, rebuttal format, scoring outputs). The rebuttal-format parenthetical mid-sentence makes it slightly run-on, but there is little waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter mutation with no output schema and no annotations, the description covers the workflow and rebuttal shape but leaves the async retrieval path and four parameters unexplained. An agent knows what the tool does and roughly when, but not how to fully parameterize or collect its result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% across 7 parameters, so the description must carry the load. It documents rebuttals well ({objection_type: text, '*': generic}) and implies script and simulation_id, but run, channel, product, and segment get no explanation at all — over half the parameters remain opaque.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: pitch a script to every simulated buyer of a simulation, collect objections, answer with rebuttals, and score the pitch plus each person. The full pipeline (pitch → objections → rebuttals → scoring) is unambiguous and distinct from siblings like market_test, which only sets up the simulation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a concrete precondition: the environment must be alive, so either run market_test with rounds first or start it. That tells the agent when the tool is callable. It doesn't name alternatives for other scenarios (e.g. email_test or website_test for non-pitch rehearsals), so it stops short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
predictB
Predictions with intervals for a run: CTR per ad, cost per lead, would-convert overall and per audience, email click rates, each scaled by the real-vs-simulated results stored so far (from backtest and calibrate) and labelled with its basis ('calibrated on N real results' or 'uncalibrated prior'); optional targets (e.g. {ctr: 2, cpl: 10}) give a probability of success. Also a 30/60/90-day adoption forecast from the best ad, its landing conversion and the word of mouth seen in the rounds.
| Name | Required | Description | Default |
|---|---|---|---|
| run | Yes | ||
| days | No | ||
| brand | No | ||
| spend | No | ||
| targets | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden and does disclose meaningful traits: predictions include intervals, are scaled by real-vs-simulated results, and are labelled with their calibration basis. It also explains that targets yield a probability of success and that an adoption forecast is produced, though it does not explicitly state read-only behavior or permission requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two dense sentences that front-load the main prediction purpose and then add output details. It is information-rich with little waste, though the semicolon-heavy structure could be easier to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with no output schema and no annotations, the description explains the kinds of predictions and their calibration labels reasonably well. It remains incomplete for the input parameters days, brand, and spend, and does not describe the return shape beyond naming metrics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for five parameters. It clarifies run and targets, including a targets example, but does not explain days, brand, or spend, leaving most parameters without semantic meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb, predict, and enumerates the resources/metrics covered: CTR per ad, cost per lead, conversion, email click rates, and adoption forecast. It is clear what the tool does, though it does not explicitly differentiate itself from siblings such as results or study.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It implies the tool should be used after backtest and calibrate by referencing their stored results as inputs, and it notes optional targets for probability of success. However, it gives no explicit when-to-use guidance versus siblings and no exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
provider_setupB
List provider choices and official key guides, inspect masked local settings, configure a local model or an existing key environment variable, or discover available models. Never paste an API key into tool arguments. ChatGPT/CLI login choices return local login commands.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| action | Yes | ||
| key_env | No | Environment variable NAME only; never the key itself | |
| base_url | No | ||
| provider | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does add real value: it warns never to paste an API key into arguments and states that local settings are inspected in masked form, plus that some providers return login commands instead of keys. What it omits is whether 'configure' mutates persistent settings, whether changes are reversible, and what 'check' actually verifies — a meaningful gap for a mutation-capable tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One dense sentence enumerating the four capabilities followed by a short, well-placed safety rule and a caveat about login commands. Every clause carries information and the security warning is front-loaded rather than buried, though the first sentence is long enough that the action-to-capability mapping takes a second pass to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and no annotations, with 5 parameters at 20% coverage, means the description has to do a lot; it covers purpose and safety well but leaves the semantics of the required 'action' values underspecified. An agent knows broadly what the tool is for and the key security rule, but could still pick the wrong action (status vs check) or omit provider/model when configuring.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 20%, so the description must compensate, and it only partly does: it implies key_env takes a variable name (reinforced by the 'existing key environment variable' phrasing and the no-paste warning) and gestures at model/base_url via 'configure a local model'. It never maps provider to the list/configure actions, and the distinction between the 'status' and 'check' actions is completely undefined, leaving the required action enum under-specified.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Uses specific verbs over a clearly bounded resource set: list provider choices/guides, inspect masked settings, configure a local model or key env var, and discover models. An agent can tell this is a provider/credential configuration tool, not a test runner like run_test or doctor. It does not name any sibling tool to distinguish overlapping areas (setup, cli, doctor), which keeps it below a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The four described modes imply when to call it (listing providers, checking local settings, configuring, discovering models) and the closing note tells the agent that ChatGPT/CLI login choices yield local login commands rather than credentials. However, there is no explicit when-not guidance and no comparison against near-neighbors like setup, doctor, or cli, so the agent must infer routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
push_contextB
While a simulation is running (or before it starts), push what people are saying today into every persona card from the next round on: a news item, a competitor's move, a real thread you just read with your browser, a price change. Each update is one or two sentences; the runner re-reads the file every round. Use it to ground the rounds in live reality.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| round | No | ||
| source | No | ||
| simulation_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses that updates flow into every persona card from the next round on, that each update is one or two sentences, and that the runner re-reads the file every round. However, it omits whether updates append or overwrite, permission/auth needs, error conditions, and what the tool returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The main action and effect are front-loaded, and the examples earn their place by making the expected input concrete. It is slightly run-on and includes a somewhat promotional closing sentence, but overall it is appropriately sized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a four-parameter mutation tool with no annotations, no output schema, and zero schema description coverage, the description gives useful runtime context but leaves significant gaps. An agent understands the intent and timing, but not fully how to populate or interpret all parameters, nor what to expect operationally.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain all four parameters. It implicitly describes 'text' as a one- or two-sentence update and gestures at 'round' via 'from the next round on', but it never explains 'simulation_id' or 'source' and gives no format expectations for optional fields.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action: pushing real-world context into persona cards from the next simulation round onward. It clearly distinguishes the tool's role from broad simulation siblings like start_live or results, though it does not explicitly name alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear timing guidance ('while a simulation is running (or before it starts)') and a concrete purpose ('ground the rounds in live reality'), with examples of acceptable inputs. It does not explicitly state when not to use it or name alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
quickstartA
Easiest start. Answer a few questions and get a ready scenario folder (brief, personas, two template captions per platform, messages, launch calendar, pitch) plus the simulated people (stops before any paid rounds). Keys: brand, product (what, for whom, price), buyers (who and where), platforms (comma list), tests (ads, messages, price, launch, pitch), budget ('300 USD'), size (small|medium|large), past_ads (absolute path, optional), docs (paths, optional). Background job; poll job_status. The result names the files to edit and the exact next command.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | ||
| no_run | No | ||
| answers | Yes | ||
| from_docs | No | documents to understand first; prefills the answers |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations to lean on, the description still discloses key behavior: it runs as a background job requiring job_status polling, and it deliberately stops before paid rounds (cost boundary). It also notes the result names files to edit and the next command. It omits whether it writes to disk, whether re-running overwrites an existing folder, and permission/auth needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the benefit ('Easiest start'), then the deliverables, then the key list, then the background-job caveat — a sensible order. The middle key enumeration is dense and comma-chained but each item carries call-relevant information; nothing is padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter tool with a nested object and no output schema, the description covers what is produced, how completion is detected (poll job_status), and what the result contains. Missing pieces are the semantics of `url` and `no_run` and any statement of runtime or failure behavior, but the core calling information is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 25% and the required `answers` parameter is an opaque nested object, but the description compensates well by enumerating its expected keys (brand, product with what/for-whom/price, buyers, platforms, tests, budget format like '300 USD', size enum, past_ads absolute path, docs). That is meaningfully more than the schema offers. The top-level `url` and `no_run` parameters remain unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States concretely what the tool produces — a scenario folder with brief, personas, two template captions per platform, messages, launch calendar, pitch, plus simulated people — and marks the boundary 'stops before any paid rounds', which separates it from paid-test siblings like market_test or run_test. It never names or contrasts against the nearest onboarding siblings (setup, understand, start_live), so differentiation is implicit rather than explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Easiest start' and 'stops before any paid rounds' imply this is the first-run entry point, and 'the exact next command' hints at a guided flow. However, no alternative tool is named and no condition for choosing this over setup, understand, or campaign_plan is given, so the agent must infer the routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
researchA
What real people say about a product, category or competitor, from public keyless sources: Hacker News stories and comments, App Store listings and recent reviews for a country storefront (in, ph, ke, us...), Reddit public search when it answers (it often blocks non-browser clients; the result says so, then use your own browser tools and add what you find as seed files), and any page URLs. Writes research.md (themes with quotes and links), research.json and voices.json (short real quotes the personas and feeds draw on) under scenarios//research. Run it before planning personas.
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| urls | No | ||
| query | Yes | ||
| country | No | ||
| sources | No | ||
| subreddits | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden well: it discloses keyless public sources, Reddit access limitations, fallback behavior, and that the tool writes research.md, research.json, and voices.json under a scenario path. It does not cover overwrite behavior or error handling beyond Reddit blocking.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but front-loads the purpose, then sources, outputs, and usage. The single long sentence with many parentheticals is somewhat run-on, but most content earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
It provides useful context for a complex tool with no annotations or output schema by naming sources, output files, and sequencing. However, it leaves key parameter meanings unexplained, which is a significant gap given 6 parameters and 0% schema description coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It partially maps query, country, and urls, but does not explain the out, sources, or subreddits parameters, leaving half the parameters undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the resource: public sentiment from Hacker News, App Store reviews, Reddit search, and page URLs, and names the output files. It is more specific than the tool name alone but does not explicitly distinguish itself from similar research-oriented siblings such as understand or study.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a clear sequencing rule ('Run it before planning personas') and a practical fallback for Reddit blocking: use browser tools and add findings as seed files. It lacks explicit when-not-to-use guidance or comparisons to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
resultsC
Key results of a run: decision, verdicts, segments, objections, messages, pricing, feed-test metrics, ROI projections, files.
| Name | Required | Description | Default |
|---|---|---|---|
| run | Yes | run name under mirofish_runs or absolute path |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden; it does add real value by enumerating the returned artifacts (decision, verdicts, objections, pricing, ROI projections, files), which is the only return-value documentation since no output schema exists. It says nothing about read-only nature, error behavior for an unfinished or missing run, or size/cost of the payload.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler. The tail is a dense comma list of return fields, which is efficient though slightly catalogue-like.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Because there is no output schema, the enumerated field list is genuinely necessary and is the description's main contribution. However, for a retrieval tool with no annotations, it omits prerequisites, failure modes, and whether results are partial for an in-progress run.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema already explains that 'run' is a run name under mirofish_runs or an absolute path. The description adds no additional meaning about the parameter, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the resource (results of a run) and enumerates what is inside it, but the verb is only implicit and there is no differentiation from siblings such as list_runs, job_status, or run_test. An agent can guess it retrieves a run's outcome, but must infer that from a noun phrase rather than a stated action.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to call this versus list_runs or job_status, and no prerequisites such as whether the run must be finished first. The only implicit guidance is the 'run' parameter name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_testB
Test anything from one spec on a prepared simulation: type = ad | message | price | offer | positioning | naming | landing | launch | pitch | email | creator_post | channel_mix; variants = the things to compare (ad creatives, message lines, prices, offer wordings, names, landing pages, launch posts with rounds, pitch scripts, emails); audience {segments, countries}; metric (default per type); budget {amount, cpm, currency}; rounds (launch). Returns one result shape for every type: each variant's metric with a 95% margin, the winner, the lift, whether it is clear or a tie, and how to read it. Background job.
| Name | Required | Description | Default |
|---|---|---|---|
| run | No | ||
| spec | Yes | ||
| brand | No | ||
| simulation_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It usefully discloses the return shape and winner/lift/tie semantics, and mentions 'Background job', but does not explain async handling, retrieval via job_status/results, permissions, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the core action and packs necessary enum-like detail into a single dense but purposeful statement. The semicolon-heavy run-on structure is slightly harder to scan, but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex nested-spec background job with no annotations or output schema, the description covers the result shape well. It still omits how to obtain or poll results, where simulation_id comes from, and how this differs from sibling test/job tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It explains many spec fields (type, variants, audience, metric, budget, rounds), but leaves simulation_id, run, and brand largely undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear verb and resource: testing variants from a spec on a prepared simulation. The long type enumeration clarifies scope, but it does not distinguish this tool from siblings like website_test, email_test, market_test, or pitch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage through 'prepared simulation' and type values, but gives no explicit when-to-use/when-not guidance or alternatives. The overlap with sibling test tools is left unresolved.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
setupA
Install or update the MarketMirror engine (git clone + uv sync). Background job; poll job_status. Needs git and uv.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does well: it discloses that execution is a background job, that completion requires polling job_status, and that git and uv must be present. It does not discuss permissions, failure modes, or whether an update is destructive to existing state.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact sentences that front-load the action and mechanism, then immediately give the async polling behavior and prerequisites. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema tool, the description covers the key operational facts: what it installs, the mechanism, that it is asynchronous, how to track completion, and prerequisites. Missing only edge-case behavior such as failure reporting or upgrade side effects.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there are no parameter semantics to explain; the baseline for 0 params is 4. The description appropriately adds no parameter detail because none exists.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (install/update) and resource (MarketMirror engine), with a parenthetical on the mechanism (git clone + uv sync). It is distinguishable from provider_setup and quickstart, though it does not explicitly name those siblings as alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clarifies when the tool applies (install or update) and gives the follow-up behavior: it is a background job, so poll job_status. Prerequisites (git and uv) are stated as a condition for use, but no explicit when-not guidance is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
start_backendA
Start the MarketMirror backend (port 5001). Configure an API-key, eligible ChatGPT sign-in, local CLI gateway, or local-model provider first.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden: it usefully discloses the bound port and a hard prerequisite, but says nothing about whether the call blocks on a long-running server, how it behaves if the backend is already running, or how success is signalled. Those are the traits an agent most needs before invoking a server-start tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with zero filler, the identity of the action front-loaded and the prerequisite following immediately. Every clause adds information an agent needs.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with no output schema the description covers purpose and prerequisite adequately, but omits what the agent should do after starting (e.g. doctor, market_test) and whether the call returns immediately or holds the session. Those gaps matter for a backend-lifecycle tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes no parameters and the schema is an empty object, so there is nothing for the description to disambiguate. Baseline 4 applies; the port mention is scope rather than parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ("Start the MarketMirror backend") and pins the concrete detail of port 5001, so an agent knows exactly which process is launched. It does not explicitly differentiate itself from siblings like start_live, provider_setup, or quickstart, which keeps it short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear precondition: one of four provider configurations must exist first (API-key, ChatGPT sign-in, local CLI gateway, local-model provider), which tells the agent the correct ordering of calls. It offers no exclusions or explicit routing to siblings, so it stops at clear context rather than full when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
start_liveB
Serve the live dashboard at http://127.0.0.1:5050/ (newest run, or a given run).
| Name | Required | Description | Default |
|---|---|---|---|
| run | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses the dashboard URL and run selection, but not whether the server blocks, runs in the background, persists after the call, requires an available port, or how to stop it.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no wasted words. The URL and run-selection behavior are both communicated compactly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool with no output schema this covers the basic purpose, but it omits important operational behavior for launching a live server. An agent knows what it does and the main parameter semantics, but not the full invocation context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single optional "run" parameter is given meaning: omit it for the newest run, or provide it for a specific run. However, the description does not specify run identifier format or examples, and schema coverage is 0%.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: "Serve the live dashboard" with an explicit URL and optional run targeting. It is clear what the tool does, but it does not explicitly distinguish itself from siblings such as start_backend or list_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage by saying it serves the newest run or a given run, which helps an agent decide whether to pass the optional parameter. However, it does not state when to use this tool versus alternatives or mention prerequisites such as whether a backend must already be running.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
studyA
Several scenarios x several seeds on the same simulated people, one defended conclusion. spec: {name, base (a market_test config object: creatives, survey, pitch, emails, rounds...), simulation_id (the people to reuse), scenarios: [{name, overrides}], seeds: [1,2,3], steps: [feed_test, rounds, survey, pitch, email_test]}. Writes study_summary.json (per-scenario mean, spread and 95% interval of every metric, deltas, winner stability, decision agreement matrix, conclusion with confidence), study_report.md and study.html under mirofish_runs//. Finished cells are reused on rerun. Background job.
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| spec | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does disclose key behavior: it writes three named artifacts under mirofish_runs/<name>/, details the contents of study_summary.json, reuses finished cells on rerun, and executes as a background job. It omits cost/runtime expectations and how to retrieve results once the job completes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Content is front-loaded and nearly every clause adds information, but it is delivered as one dense run-on with an inline schema sketch, making it hard to scan. Some structuring (separate sentences for what it does, the spec shape, and what it writes) would cost nothing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations, no output schema, and a nested spec object at 0% schema coverage, the description is thorough about spec internals, output artifacts, and rerun behavior. It still leaves an agent without guidance on the 'out' parameter or on how a finished background job is collected.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does: every key of the required nested spec (name, base, simulation_id, scenarios/overrides, seeds, steps) is enumerated and given meaning, with base explicitly typed as a market_test config. The optional 'out' parameter is never mentioned, which is the one real gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb/resource: a study runner that crosses several scenarios with several seeds over the same simulated people and yields 'one defended conclusion.' It distinguishes itself from the single-run sibling by defining base as 'a market_test config object,' but it never explicitly contrasts with market_test, run_test, or predict.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Background job' implies polling via job_status and 'Finished cells are reused on rerun' implies resumability, so usage is inferable. However, there is no explicit when-to-use-this-vs-market_test guidance, no statement of prerequisites, and no exclusion of alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
understandA
Step one for any product: read the user's documents (absolute paths to .md/.txt), a landing page URL and/or an ads export into one product profile (what it is, for whom, pricing, offer, proofs, differentiators, competitors, channels, likely objections, brand voice, KPIs, unknowns). Every later step uses it. Synchronous, one Claude call. Writes product_profile.json; pass project instead of docs to profile an existing project (also seeds the market map before the graph build).
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| url | No | ||
| docs | No | ||
| name | No | ||
| project | No | ||
| ads_file | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full load — and it discloses real behavioral traits: synchronous execution, a single Claude call, a file side effect (writes product_profile.json), and a dependent side effect (seeds the market map before the graph build). It omits overwrite/error behavior and what happens when no inputs are supplied.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded in the first clause and the dense parenthetical enumerating profile fields is informative rather than filler. The closing sentence bundles three unrelated facts (output file, project mode, market-map seeding), which slightly hurts scanability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations, no output schema, and six undocumented parameters, the description is the only source of truth. It covers input modes, the output artifact, and key side effects well, but the two unexplained parameters and the absent failure/overwrite semantics leave a gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It explains `docs` (absolute .md/.txt paths), `url` (landing page), `ads_file` (ads export) and `project` (existing project), but never mentions `out` or `name`, leaving two of six parameters undocumented anywhere.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('read ... into one product profile') and enumerates exactly what the profile contains, so the agent knows what the tool produces. 'Step one for any product' positions it relative to the later-step siblings, though it never names which sibling to use next.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states the entry condition ('step one for any product') and an explicit alternative invocation mode ('pass `project` instead of docs to profile an existing project'). It doesn't warn about when not to use it, e.g. when a project has already been profiled, or how it differs from `research` or `quickstart`.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
website_testB
Run prepared synthetic visitors on a real website URL. Actions and screenshots are observed; only supplied DOM checks verify goals. Cart changes are enabled by cart/checkout goals; signup submission requires explicit test data and opt-in. No payments/orders. Background job: poll job_status.
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | ||
| spec | Yes | ||
| simulation_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and discloses important behavior: actions and screenshots are observed, only supplied DOM checks verify goals, cart changes require cart/checkout goals, signup requires explicit test data and opt-in, and no payments/orders occur. It omits permissions, result retrieval, and side-effect details beyond those constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded, with the core action first and constraints following. It is dense but earns most of its sentences; a slightly clearer structure around background polling and results retrieval would help.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a complex nested-schema tool with no annotations and no output schema. The description covers key safety constraints and background polling, but it does not explain the nested spec parameters, the meaning of simulation_id, or how to retrieve results beyond polling job_status.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is effectively 0% for most parameters, and the description explains only a few concepts: checks, cart/checkout goals, test data/opt-in, and job polling. It leaves major parameters such as simulation_id, spec.url, goal, task, device, max_steps, observation_mode, and resource limits unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Run prepared synthetic visitors on a real website URL.' It clearly conveys a browser-based synthetic-user test, though it does not explicitly differentiate itself from siblings like email_test, market_test, or run_test.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives useful operational context: this is a background job and you should poll job_status. It also specifies gating conditions for cart changes and signup submission. However, it does not say when to choose this tool over email_test, market_test, or run_test.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
27 tool updates
v0.4.0- First observed
actuals_import - First observed
ads_import - First observed
backtest - First observed
calibrate - First observed
campaign_plan - First observed
cli - First observed
doctor - First observed
email_test - First observed
holdout_backtest - First observed
ideas - First observed
job_status - First observed
list_runs - First observed
market_test - First observed
pitch - First observed
predict - First observed
provider_setup - First observed
push_context - First observed
quickstart - First observed
research - First observed
results - First observed
run_test - First observed
setup - First observed
start_backend - First observed
start_live - First observed
study - First observed
understand - First observed
website_test
TDQS
Scored across 27 tools
The tools cluster into setup, ingestion, simulation, testing, and analysis, but several boundaries are fuzzy: market_test vs run_test vs study all orchestrate simulated tests, backtest vs holdout_backtest both score against real results, and ads_import vs actuals_import both ingest exports. Multiple entry points (understand, quickstart, research) also require reading descriptions to know which to use first. The descriptions do help, but an agent could easily misselect among the test and backtest tools.
All names are snake_case, which is consistent at the casing level. However, the semantic pattern is mixed: some are noun-only (ideas, results, pitch, study), some are verb-only (doctor, setup, calibrate, predict), and others follow verb_noun (run_test, list_runs, start_backend). It is readable but not a predictable convention.
At 27 tools, the surface is heavy for the apparent scope and includes many advanced or overlapping entry points (market_test, run_test, study, email_test, website_test, pitch, cli). While each has some role, the count exceeds the well-scoped 3-15 range and increases selection cost.
The lifecycle is broadly covered: setup/doctor, document understanding, research, ad/actuals ingestion, persona simulation, modular and all-in-one tests, email and website tests, backtests, calibration, prediction, dashboards, and raw CLI access. Minor gaps remain around managing or deleting runs/projects, editing personas, and exporting reports, but core workflows are well supported.
Maintenance
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
Hosted Amazon Seller and Vendor MCP server for Claude, ChatGPT, Cursor, Codex, Gemini, Copilot.
Hosted MCP for e-commerce: live product catalog, stock, and pricing for AI agents.
Related MCP Servers
- AlicenseAqualityCmaintenanceAn MCP server that enables users to query, compare, and synthesize responses from multiple local and cloud LLMs simultaneously using existing subscriptions. It provides tools for parallel model evaluation, consensus polling with an LLM-as-judge, and response synthesis across different model providers.834 npm15MIT

mock-llm-mcpofficial
AlicenseNot gradedqualityDmaintenanceMCP server to mock OpenAI, Anthropic, and Google Gemini responses for testing AI integrations without real API keys or token spend.MIT
AdsTurbo Creative MCPofficial
AlicenseAqualityDmaintenanceA local-only MCP server for AI video ad planning. It helps AI agents build video ad briefs, hooks, UGC scripts, storyboards, variation plans, script reviews, and AdsTurbo-ready prompts without calling AdsTurbo internal services.721 npm2MIT- AlicenseAqualityBmaintenanceLocal MCP server that plans, generates, and assembles production assets (images, audio, video) through multi-agent personas and official APIs, with free-tier budget guard.1415MIT