Skip to main content
Glama

Pulse

A screen map and verified-workflow store for computer-use agents.

Pulse records what a page or desktop looks like. It connects screens with transitions it watched succeed. Then it replays a whole route in one call.

Python MCP server. No runtime dependencies.

The problem

An agent driving a UI pays the same cost on every step. Observe, decide, click, observe again to confirm. A route it solved yesterday is solved from scratch today.

Pulse keeps the map between runs. The learned route replays as one call, and the replay still verifies every state it crosses.

flowchart LR
  O[observe] --> M[("map<br/>screens + anchors")]
  M --> T[("transitions<br/>verified edges")]
  T --> R[shortest route]
  R --> P[replay]
  P -->|new screens| M

Related MCP server: dsh-computer-use

With Pulse and without

flowchart TB
  subgraph W["Without Pulse: every step re-observes"]
    direction LR
    W1[observe] --> W2[click] --> W3[observe] --> W4[click] --> W5[observe] --> W6[click] --> W7[observe]
  end
  subgraph P["With Pulse: one verified route"]
    direction LR
    P1["replay call"] --> P2{"verify each state"} --> P3["final state returned"]
  end
  W ~~~ P

The saving tracks how much observation a route removes. Six guarded clicks become one call. A single click saves one screenshot digest and nothing else.

Measured results

Every number below comes from docs/benchmark.md, on local fixtures and public sites, on 2026-10-24.

One call against many

Workflow

Baseline

Replay

Saved

Evidence

6 clicks, in process

4,328 ms

1,177 ms

72.8%

3 trials each

3 clicks, in process

309.0 ms

95.9 ms

69.0%

7 trials each

2 desktop clicks

3,364.9 ms

1,047.7 ms

68.9%

7 trials each

4 transitions, model in loop

42.76 s

17.39 s

59.3%

3 paired trials

Whole workflows, live sites

Workflow

Calls

Baseline

Replay

Change

Add two elements

2 → 1

751 ms

177 ms

−76.4%

Toggle a checkbox

1 → 1

440 ms

231 ms

−47.4%

Scroll a long page

1 → 1

364 ms

275 ms

−24.4%

Complete a todo

3 → 3

903 ms

773 ms

−14.5%

Nine workflows through each harness

Harness

Baseline p50

Replay p50

Speed increase

Runs

Claude Code

17.83 s

12.49 s

42.8%

35/35 correct

Codex

34.55 s

23.00 s

50.2%

36/36 correct

The six-click workflow alone speeds up 278.2% through Claude Code and 276.2% through Codex. Both harnesses hit that target on sequential work. Neither reaches it on random or single-step requests.

The same runs inside agent desktops

Each harness ran in its own pane, on its own agent desktop, with its own Chromium profile, CDP port and Pulse state per trial. The host focus never moved.

Workflow

Harness

Baseline p50

Replay p50

Speed increase

Runs

Six clicks

Claude Code

49.97 s

17.38 s

187.5%

4/4 correct

Six clicks

Codex

85.03 s

23.35 s

264.2%

4/4 correct

Two categories

Claude Code

19.73 s

14.57 s

35.3%

4/4 correct

Two categories

Codex

36.45 s

26.15 s

39.4%

4/4 correct

Nine workflows ran in the same conditions. Nine workflows moved the overall p50 from 28.94 s to 18.64 s through Claude Code, a 55.3% speed increase. Nine workflows moved 39.40 s to 26.43 s through Codex, 49.1%. Nine workflows include six clicks at 354.1% and 256.4%, two Claude regressions, and two Codex refusals that stopped on a stale start screen rather than acting. Every trial recorded deepseek/deepseek-v4.1-flash-ultrafast in its own row. The per-workflow table is in docs/benchmark.md.

xychart-beta
  title "p50 wall time, nine live workflows"
  x-axis ["baseline", "replay"]
  y-axis "seconds" 0 --> 40
  bar [17.83, 12.49]

Good cases

Pulse pays for itself when a route removes observation.

Case

Why it gains

Measured

Many sequential steps

one guarded step leaves per transition

6 clicks: 72.8% saved in process, 278.2% faster in Claude

A route you run again

learning cost spreads over every rerun

nine workflows: 42.8% and 50.2%

Deterministic DOM

screen ids match across runs

most workflows 2/2 or 3/3

Desktop with a settled tree

accessibility map replaces screenshots

68.9% saved over 2 clicks

Base cases

Pulse gains little or nothing here. Each limit has a reason.

Case

Why it does not gain

Measured

A single step

both paths observe twice, so a route saves one digest

scroll: 25.0% slower in Claude

A short workflow

the CLI launch floor, 5.7 s Claude and 11.4 s Codex, dominates

two clicks: 23.3% faster

A page typed value reappears on

the page renders the value as mapped text

TodoMVC

Lazy-loading content

settled height differs per run

replay stopped 1/3, then 3/3 after a settle wait

Action reported only in a <p>

no mapped anchor changes, so no edge forms

demoqa buttons

Replay pays one settle wait per step. Replay pays 631 ms for the call wrapping a 4.6 ms read. Replay needs that wait to record an async step correctly, so the cost is a correctness decision rather than a tuning knob.

Install

python3 -m pip install -e .
pulse-mcp

The default data directory is ~/dev/pulse-data when ~/dev exists. Otherwise it is $XDG_STATE_HOME/pulse, or ~/.local/state/pulse. Set PULSE_STATE_DIR to choose another directory.

Connect

Pulse speaks MCP over stdio. pulse-mcp reads JSON-RPC on stdin and writes it on stdout, so each client launches its own child process. Pulse opens no port and runs nothing between sessions.

claude mcp add --scope user pulse -- pulse-mcp
# ~/.codex/config.toml
[mcp_servers.pulse]
command = "pulse-mcp"

Restart the harness after adding the server. Both harnesses expose the same tools.

Tools

Tool

Purpose

pulse_map_record_screen

Save an observation, return its grid

pulse_map_record_transition

Save an edge, verified or not

pulse_map_get

Read a map, or one screen with its grid

pulse_map_route

Shortest route over verified edges

pulse_routine_save / list / get

Save and read routine candidates

pulse_browser_observe

Map visible controls in a local Chromium tab

pulse_browser_targets

List tabs on one CDP port, and which a call picks

pulse_browser_click

Click a mapped control after a stale-screen check

pulse_browser_fill

Type into one mapped field, guarded, no implicit submit

pulse_browser_key

Send enter or escape to the focused element

pulse_browser_scroll

Scroll a bounded delta and re-map

pulse_browser_replay

Run a verified route in one call

pulse_desktop_observe

Map a Linux desktop from accessibility and screenshots

pulse_desktop_click

Click a mapped desktop control, guarded

pulse_desktop_replay

Run a verified desktop route

Anchor bounds are normalized. x, y, width and height each use the viewport as a 0–1 scale, and an anchor sits in the grid cell holding its center. An edge becomes route-eligible only after it is seen to change screen state.

Start a browser

chromium --headless=new --no-sandbox --remote-debugging-port=9222 \
  --user-data-dir=/tmp/pulse-chromium about:blank

The default port is 9222. Tools can pass another local port, or set PULSE_CDP_PORT. Connect only a browser profile you intend an agent to control.

A call without target_url takes the newest page target, which may not be the page under test. A call with target_url stays on the tab it names. A call should follow pulse_browser_targets when one port serves several tabs, because that tool reports what the port holds and which tab a call would pick. A call that captures a stalled background tab reports no digest, and the observation still returns its anchors.

What a map holds

Holds

Never holds

roles, labels, selected state

form values

normalized bounds, viewport size

screenshot bytes

screenshot digest

URL query or fragment

Map files are written atomically with owner-only permissions. A fill keeps its value out of every map, tool result and saved routine. A fill reports only the length it typed. A fill refuses password, file, hidden, checkbox, radio, disabled and read-only fields. A fill confirms by reading the field back, and sends no key, so it cannot submit.

Two rules govern focus and line endings:

  • A fill drops focus once it confirms, because a focused field's text reaches the next observation's anchor labels. Pass keep_focus before pulse_browser_key.

  • A single-line field loses a trailing newline, so the entry is trimmed and confirmed against the trimmed text.

A scroll position takes part in the screen id. Two positions in one long page therefore have separate ids, and a stale-screen check cannot pass wrongly.

The fill guard has one leak. A value reaches the map when the page displays it. A value arrives as an anchor label like any other text. A value cannot be redacted without dropping real page content. A value never belongs in a secret field the page shows, and TodoMVC shows exactly that: a typed title becomes the row's checkbox label.

Desktop

The desktop adapter reads computer-use-linux accessibility snapshots and screenshots. The desktop adapter places accessible elements on the same normalized grid. The desktop adapter needs computer-use-linux mcp on PATH, and PULSE_COMPUTER_USE_BIN picks a specific binary. The desktop adapter calls Tesseract only to place a control whose accessibility bounds are missing. Browser routes and desktop routes share one planner.

Limits

Pulse does not execute saved routines in this version. Routine steps reject literal values, so a step carries an input_ref and a future driver supplies the value.

A paragraph is not mapped. A page that reports an action only in a <p> changes no screen id, so the map looks unchanged and no route learns.

Development

PYTHONPATH=src python3 -m unittest discover -s tests -v
python3 -m compileall -q src tests

Results, the four defects live sites exposed, and the scope of each number are in docs/benchmark.md.

Available Tools

17 tools
pulse_browser_clickB

Click a mapped browser control only if the current screen still matches the expected map state. Write and irreversible controls require explicit authorization.

Inputs: anchor_id (required), expected_screen (required), port, app_id, +2 more.

pulse_browser_fillA

Type a value into one mapped browser field. The value applies to this call only, is never stored or returned, and never submits the form. Requires authorized_effects:["write"]. Password, file, hidden, checkbox, radio, disabled and read-only fields are refused. Set keep_focus when a fill is followed by pulse_browser_key, because a fill drops focus by default to keep the value out of the map.

Inputs: value (required), anchor_id (required), expected_screen (required), authorized_effects (required), +4 more.

pulse_browser_keyB

Send one allowlisted key, enter or escape, to the focused element on the expected screen. Enter submits the form that holds the focused field. Requires authorized_effects:["write"].

Inputs: key (required), expected_screen (required), authorized_effects (required), port, +2 more.

pulse_browser_observeB

Observe the active local Chromium page, map visible DOM controls into the spatial grid and store the screen.

Inputs: port, app_id, target_url.

pulse_browser_replayB

Execute a previously verified route in one browser call. Each step checks its expected screen and stops on divergence. A completed route returns the final screen_id, title, viewport, anchor_count, anchors and screenshot_sha256, so no confirmation observe is needed. Write and irreversible effects require explicit authorization.

Inputs: start (required), target (required), port, app_id, +2 more.

pulse_browser_scrollC

Scroll the current browser viewport by a bounded pixel delta and map the resulting screen. The scroll acts only on the viewport of the expected mapped screen.

Inputs: expected_screen (required), dx, dy, port, +2 more.

pulse_browser_targetsA

List the page tabs one CDP port serves and report which one a call would choose. Pass target_url on other browser calls to pin the page under test when a port holds more than one tab.

Inputs: port, target_url.

pulse_desktop_clickA

Click an anchored desktop control only when the live desktop still matches the expected mapped screen. Write effects require explicit authorization.

Inputs: anchor_id (required), expected_screen (required), app_id, application, +1 more.

pulse_desktop_observeC

Capture the active Linux desktop through computer-use-linux, map accessibility controls to a normalized screen grid, and use OCR for anchors when accessibility bounds are missing.

Inputs: app_id, application.

pulse_desktop_replayB

Execute a verified desktop route in one call. Each step checks the expected mapped screen and stops on divergence. Write effects require explicit authorization.

Inputs: start (required), target (required), app_id, application, +1 more.

pulse_map_getB

Read a stored screen map, optionally including one screen's full grid.

Inputs: app_id (required), screen_id.

pulse_map_record_screenC

Record a screen observation with normalized anchors and return its grid.

Inputs: url (required), title (required), app_id (required), anchors (required), +4 more.

pulse_map_record_transitionC

Record a transition. Only verified transitions are used by route planning.

Inputs: app_id (required), source (required), target (required), verified (required), +1 more.

pulse_map_routeC

Find the shortest verified route from a mapped screen to a screen or anchor label.

Inputs: start (required), app_id (required), target (required).

pulse_routine_getB

Read a saved routine candidate by id.

Inputs: routine_id (required).

pulse_routine_listB

List saved routine candidates, optionally filtered by application.

Inputs: app_id.

pulse_routine_saveC

Save a successfully verified routine as a reviewable candidate; this does not execute it.

Inputs: id (required), name (required), steps (required), app_id (required), +2 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 17 tools

    Disambiguation4/5

    The four domain prefixes (map, browser, desktop, routine) plus action suffixes make most tools clearly separable: browser_click vs desktop_click vs map_get vs map_route are all obviously different operations. The main soft spots are pulse_browser_observe vs pulse_map_record_screen (both capture a screen into the grid, differing mainly in source and side effects) and pulse_browser_replay vs pulse_map_route, which an agent could confuse as plan-vs-execute.

    Naming Consistency4/5

    Every tool follows a pulse_<domain>_<action> pattern with a stable set of domain prefixes, which makes grouping predictable. The only deviation is argument ordering inside the name (map_get/record_screen are noun-then-verb while browser_observe/desktop_click are verb-only), but this is minor and readable.

    Tool Count4/5

    17 tools is slightly above the comfortable range but justified by the four distinct sub-surfaces (mapping, browser, desktop, routines), each of which earns two to seven tools. Nothing appears gratuitous, though the map and routine groups are thin enough that some consolidation might be possible.

    Completeness3/5

    The browser surface is rich (observe, targets, click, fill, key, scroll, replay) but the desktop surface only offers observe/click/replay with no direct fill, key, or scroll primitive, and map/routine groups lack listing of screens and any delete/update for saved routines or transitions. These gaps force agents to work around missing operations or rely entirely on replay for desktop input.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      C
      quality
      C
      maintenance
      Enables agent-native desktop control by driving mouse, keyboard, and interface elements through accessibility-first semantic operations, with fallback to screen coordinates when accessibility cannot reach a target.
      30
      MIT
    • A
      license
      Not graded
      quality
      F
      maintenance
      Enables AI agents to record desktop workflows from human demonstrations, save them as validated semantic skills, and replay them in an isolated Linux sandbox with deterministic verification and evidence-driven self-healing.
      1
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables an agent to observe and operate desktop apps and browsers through an accessibility-tree-first view, with clicking, typing, scrolling, dragging and waiting tools. Screenshot handling, coordinate mapping and fail-closed safety rules (risk confirmation, deny-listed apps, stale-observation refusal) are enforced inside the server.
      9
      Apache 2.0