multi-modal-jev-browser

A browser agent that asks a small, cheap decision model what to do next and calls a vision LLM only when that model is unsure. It runs as a CLI, a Python library and an MCP server, so Claude Code, opencode, Cursor and Codex can drive a real browser with one line of config.

One form, three decision models, the same browser layer and the same loop. Left: Jev reading the element list, with an LLM fallback: 8 s and $0.0001. Middle: Clef reading the numbered screenshot: 13 s and $0.0053. Right: DeepSeek v4.1 Flash deciding every step: 35 s and $0.0019. All three submit the right values. Real time, played 2.5x faster. Reproduce it with examples/compare_deciders.py.

Why it is cheap

Most browser agents give every click to a large LLM: read the page, think, call a tool, repeat. mmjb splits the job.

• A decision model answers typed multiple-choice and yes/no questions with probabilities, one call per step. Jev costs $0.000015 a call and answers in about a second. Clef, which also reads the screenshot, costs under a tenth of a cent a step.

• An ordinary vision LLM takes one step only when the decision model is unsure, claims nothing can move, or the page stops changing.

On the benchmark below, a Jev run needed the LLM fallback in none of 15 tasks.

The cost and time rows are the measured cost and time per step multiplied by the number of steps. The full tables, the method and the reproduction commands are under Results.

Related MCP server: Cloudflare Playwright MCP

At a glance

Pick a decision model

How a step works

  page            observe            one call            act             say what happened
  ──────  ────────────────────  ────────────────────  ─────────────  ────────────────────────
  tab  ─▶  numbered screenshot  ─▶  decision model    ─▶  click /     ─▶  "same url; the page
          + element list            operation            type /          text changed by +27
          + page text               click target         select /        characters"
          + facts                   goal_done            scroll          + the next step record
          + last 8 effects          no_move              back
                                     fact_for
                                          │
                              margin < 0.1, or no move,
                              or 3 dead clicks in a row
                                          ▼
                                 vision LLM fallback

Quick start

git clone https://github.com/foklepoint/multi-modal-jev-browser
cd multi-modal-jev-browser
uv venv --python 3.12
uv pip install -e ".[dev]"
uv run playwright install chromium
export CLOUDFLARE_API_TOKEN=...      # or OPENROUTER_API_KEY, see the table below
export CLOUDFLARE_ACCOUNT_ID=...
uv run mmjb doctor
uv run mmjb run --url https://example.com/contact \
  --goal "Fill in the contact form with the facts and send it." \
  --fact name="Ada Lovelace" --fact message="I would like to hear more." \
  --allow-writes

mmjb doctor checks Python, the browser, every decision model and the fallback, and prints the exact fix for anything that fails. It exits 0 only when the browser works and at least one decision model answers.

Without a clone, for an MCP server:

uvx --from git+https://github.com/foklepoint/multi-modal-jev-browser mmjb-mcp

MCP setup

The server keeps one browser open for the life of the session, so look, click, type and the rest all work on the same page.

claude mcp add mmjb --env CLOUDFLARE_API_TOKEN=your_token --env CLOUDFLARE_ACCOUNT_ID=your_account_id -- uvx --from git+https://github.com/foklepoint/multi-modal-jev-browser mmjb-mcp

Copy-paste config for opencode (opencode.json), for Cursor and Codex, and a generic mcpServers file are in examples/. Every one of them uses the same uvx --from git+https://github.com/foklepoint/multi-modal-jev-browser mmjb-mcp command.

Tools:

Choosing and swapping the models

The decision model is the cheap one that makes the decisions. The fallback model is the expensive vision LLM that takes one step when the decision model is unsure. Swap either with one environment variable.

With nothing set, the choice is Clef when the Cloudflare variables exist, then Jev when OPENROUTER_API_KEY exists, then a LiteLLM model.

The fallback model is MMJB_FALLBACK_MODEL, default openrouter/deepseek/deepseek-v4.1-flash. Any LiteLLM model string works.

The cost of a step is one decision-model call. In practice most simple forms finish in six to ten steps with no fallback call at all.

Command line

mmjb run --goal "..." --url "..." --fact key=value --fact other=value \
         [--allow-writes] [--max-steps 40] [--budget 300] [--json] [--out shot.jpg]

mmjb look --url "..." [--out shot.jpg]     # one numbered screenshot and the element list
mmjb doctor                                 # check the setup, exit 0 only when it works

mmjb run --json prints one JSON object on stdout and everything else on stderr, so it can be piped straight into another tool. Exit codes: 0 verified, 2 blocked, 3 unverified, 4 step limit, 1 error.

Using it as a library

from mmjb import Agent

agent = Agent(
    goal="Fill in the contact form and send it",
    start_url="https://example.com/contact",
    facts={"name": "Ada Lovelace", "message": "I would like to hear more."},
    allow_writes=True,
)
result = await agent.run()
print(result.outcome, len(result.steps), f"${result.cost:.4f}", result.seconds)

from mmjb import Agent, Result, CloudflareClef, TypesafeJev, LLMDecider also works, and every tool and option has a one-sentence docstring that reads well as an MCP tool description.

Writing your own decision model

A class with one async method. Twenty lines is a working model.

from mmjb.deciders import Answer


class MyDecider:
    name = "mine"

    async def decide(self, state: dict, questions: dict, images: list[str]) -> dict[str, Answer]:
        ...
        return {"operation": Answer(key="operation", kind="choice", choice="CLICK",
                                    probabilities={"CLICK": 0.8, "WAIT": 0.2}, confidence=0.8)}

MMJB_DECIDER=mypackage.decider:MyDecider mmjb run --goal "..." --url "..."

The same works for a fallback (MMJB_FALLBACK=mypackage.fallback:MyFallback), a safety policy (MMJB_POLICY=mypackage.policies:allow_something) and a step hook (Agent(on_step=...)). Installed packages can publish deciders and fallbacks under the mmjb.deciders and mmjb.fallbacks entry point groups, so pip install some-plugin adds one with no change to this repository. Worked examples for all four are in docs/extending.md and in examples/.

Limits

It cannot do these things, and there is no point pretending otherwise:

• custom dropdowns that need keyboard handling to open, because it only sees the DOM controls a button click would reach

• canvas applications, games and anything drawn in pixels rather than in HTML

• CAPTCHAs and browser verification challenges

• drag and drop

• anything behind a login it has no credentials for

• reading a file input the page never shows as a control

• file uploads are implemented (a file: fact and set_input_files) but only covered by the safety tests, not by an end-to-end run

It also works in the viewport. Content below the fold is listed as an offscreen control and reached by scrolling to it, one step at a time.

Safety

• allow_writes is false by default. Nothing is typed, selected or uploaded, and no submit, send, save, post or publish button is clicked, until it is on.

• Even with allow_writes on, delete, remove, log out, pay, buy, upgrade, subscribe and checkout are refused unless the goal itself contains that word.

• A custom policy can replace these rules entirely.

• Page text is untrusted data, never instructions, and the rules text handed to every model says so.

• Keys are read only from environment variables. Nothing is ever written to a file.

Tests

uv run playwright install chromium
uv run pytest -q

The suite is offline: a local test site is served by http.server and a scripted decision model drives the real loop, so the tests make no network call and need no API key. Tests marked live run the same site against a real decision model and skip themselves unless the keys are set:

CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ACCOUNT_ID=... uv run pytest -m live -q -s

Results

Five form tasks of rising difficulty, served from examples/bench_site: a contact form, a demo request that opens in a new tab and keeps its submit button disabled until the form is valid, a three step wizard, a long form that needs scrolling, and a sign up with a 40 country dropdown. Every page posts what was submitted to a local server, and a run counts as correct only when the exact expected values arrived, whatever the agent claims.

Accuracy, speed and cost on the benchmark

Correct submissions by task

Median steps and seconds by task

What one step costs

The same models on longer tasks (measured cost per step, multiplied)

A cost per step measured above, times the number of steps. Real long tasks also hit more unsure steps, so treat the larger columns as a floor for the cheaper models.

Time on longer tasks

Cost compared with GPT-6 Luna only

How to read these numbers

• The pages are local and were written for this benchmark. Real sites are messier, so expect more unsure steps and more fallback calls there than here.

• The LLM-only rows were run on the shortest tasks only, because they are slow. Their cost and time per step come from those runs.

• Costs: Jev is $0.000015 a call at list price, Clef is the input tokens in the response times $0.24 per million, and the LLM rows and any fallback calls are the cost LiteLLM reports.

• The tables for longer tasks multiply the measured cost and time per step by the number of steps. They assume the same share of unsure steps as the benchmark.

• The first pass of this benchmark found that the done check did not recognise "Your account was created" as a confirmation. The word list was extended and the sign up task was run again; the tables use the shipped code.

Reproduce:

uv run python examples/bench.py run --label jev --decider jev --repeats 3
uv run python examples/bench.py run --label clef --decider clef --repeats 3
uv run python examples/bench.py run --label deepseek --decider llm --model openrouter/deepseek/deepseek-v4.1-flash --repeats 3
uv run python examples/bench.py report tmp/bench/*.jsonl
uv run python examples/compare_deciders.py --runs 3        # the three runs behind the GIF

Credits

• Jev by TypeSafe, for the decision-model idea and the shape of the questions

• Clef by Cloudflare, on Workers AI

• inspired by browser-use and jev-ultrafast

• LiteLLM, for the model-agnostic fallback

• Playwright, for the browser

This project is unofficial. It is not affiliated with TypeSafe or Cloudflare.

Licence

MIT. See LICENSE.