evolution-computer-use
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
evolution-computer-use
English | 简体中文
Self-evolving computer use for any agent.
Extracted from dsh-world 0.8, where it was "her own browser and the operations she learned", and freed from the DSH host. How procedures are stored, replayed and repaired was chosen by two controlled experiments run while building dsh-world; see Experiments.
A browser of its own: the system's Edge (or Chrome) on a separate profile, driven over the DevTools Protocol (CDP). It never uses your mouse or keyboard and never touches your everyday browser.
The user's computer (Windows, only when the user turns it on):
background mode posts input to one window, without moving the mouse or taking the focus;
foreground mode works the whole screen with the real mouse and keyboard.
A procedure library: once the agent has done something with page primitives or desktop actions, it saves it as a procedure. Next time it replays it without the model, with no step-by-step exploration.
When a site redesign breaks a procedure, the replay stops at the failing step and says why. The agent finishes the task by hand and saves a new version under the same name.
When the new version fails, the old ones are tried automatically.
Every agent using this plugin can share the same library.
The plugin contains no code for any particular platform (Bilibili, WeChat, …). How a site or program is used is for the agent to find out, and it is saved only once it works.
No runtime dependencies: the browser side uses only Node's built-in WebSocket and fetch. The desktop side is one PowerShell script (src/desktop/win32.ps1) using Windows' own APIs.
Language: the agent-facing text (tool descriptions, results, error messages, the usage instructions sent at initialization) is in Chinese, as in dsh-world. Current models follow it fine whatever language the user speaks; the documentation is in English first.
Setup
Requires Node.js 22.16 or later.
git clone https://github.com/chenkezhen480/evolution-computer-use.git D:\tools\evolution-computer-use
cd D:\tools\evolution-computer-use
npm install
npm run build # builds lib/; the configurations below run lib/cli.jsReplace the paths below with your own. --desktop defaults to off (the agent's own browser only); to let the agent use your computer, pass background or foreground, as described in The user's computer.
Claude Code
As a plugin (recommended; runs from source, no build needed):
/plugin marketplace add chenkezhen480/evolution-computer-use
/plugin install evolution-computer-useA local clone works too: give its path to marketplace add.
Or add only the MCP server:
claude mcp add evolution-computer-use -- node D:/tools/evolution-computer-use/lib/cli.js --desktop backgroundCodex CLI
~/.codex/config.toml:
[mcp_servers.evolution-computer-use]
command = "node"
args = ["D:/tools/evolution-computer-use/lib/cli.js", "--desktop", "background"]
# Recording up to two minutes of video, or waiting in the browser queue, can exceed the default 60 s
tool_timeout_sec = 180Cursor, Cline, Windsurf, Claude Desktop and others
These clients take the mcpServers form (.cursor/mcp.json for Cursor, claude_desktop_config.json for Claude Desktop):
{
"mcpServers": {
"evolution-computer-use": {
"command": "node",
"args": ["D:/tools/evolution-computer-use/lib/cli.js", "--desktop", "background"]
}
}
}Any other MCP agent
Any stdio MCP client works.
Command:
node <path>/lib/cli.js [options].Protocol versions: 2025-06-18, 2025-03-26 and 2024-11-05.
Instructions: the usage guide is sent in
instructionsat initialization.Images: screenshots come back as MCP
imagecontent.page_shotanddesktopare only useful if the client passes images to the model.
Embedding in JS/TS
Hosts that do not speak MCP (your own agent loop, a DSH plugin) use the library directly:
import { createComputerUse, INSTRUCTIONS } from 'evolution-computer-use';
const computer = createComputerUse({
dataDir: 'D:/agent-data/computer', // holds procedures/, browser/ and messages.jsonl
name: 'Helper', // the name on the banner while it acts on the user's computer
onToolsChanged: () => { /* desktop control was switched: register the tools again */ },
});
computer.setDesktop(true); // only once the user agreed
// 1. Put INSTRUCTIONS in the system prompt. 2. Give the tools to the model:
const tools = computer.tools().map(({ name, description, inputSchema }) => ({ name, description, input_schema: inputSchema }));
// 3. When the model calls a tool:
const result = await computer.call(name, args, signal); // { text, images?: [{ data: Buffer, mediaType, caption? }] }Other methods:
setForeground(on): foreground mode; shows a UAC prompt.halt()/release(): stop the agent and release it, for hosts with a stop button of their own.routines(): run the routine procedures.status().show(): brings the browser window forward.stop().
The messages option takes your own MessageSink, to file the messages procedures read into the host's storage.
Related MCP server: EchoPath MCP Server
Command line
evolution-computer-use [serve] [options] run as an MCP server over stdio
evolution-computer-use resume [options] release the agent after the user pressed "stop"
evolution-computer-use routines [options] run the routine procedures once and print the results (for schedulers)
evolution-computer-use status [options] list learned procedures and anything brokenOption | Meaning |
| Data folder, default |
| Procedure library, default |
| Browser profile, default |
| Desktop control, default |
| The name on the banner while the agent acts, default |
| Programs the inbox does not report, comma-separated |
| Do not watch the taskbar even with desktop control on |
| Do not file what procedures read into |
Faults and recoveries go to stderr (MCP clients usually keep it in their own logs). The agent also sees them when it calls computer_use without arguments.
Tools
Tool | What it does |
| Opens an address in the agent's browser and returns the page (as |
| Reads the page: one line per actionable element, |
| Clicks an element by ref; refuses when something covers it |
| Types into an input by ref, replacing its content; |
| Presses one key |
| Scrolls about a screen up or down, optionally inside an element |
| Waits for some text (up to 60 s), or checks that a ref still exists |
| A screenshot of the page, for when the text is not enough |
| Runs JS in the page; 10 s timeout, result cut to 2000 characters |
| Names the program or site about to be used ( |
| Runs a learned procedure: |
| Saves working steps as a procedure; without |
| Writes a category's notes (which program or site, addresses and aliases, caveats), up to 1000 characters |
| Runs every |
| Acts on the user's computer (only while desktop control is on); see below |
| Reads the inbox (only while the inbox is on) |
The tool list is fixed: learning something new never changes it, so saving a procedure never invalidates the host's prompt cache. Only the user's switches make desktop and inbox_read appear or disappear; the server then sends notifications/tools/list_changed. Results are short: page reads are paged, script results cut, and screenshots never attached unasked.
How it evolves
Find:
computer_use("微信").The target is matched only against category names and their notes, word by word and case-insensitively.
One word containing the other counts as a match, and so do words sharing at least 60% of their letter pairs ("bilbili" finds "bilibili").
Procedure names do not take part: a "send message" for Bilibili and one for WeChat are different things.
One category comes back at a time. When several match equally well, the agent is asked which one; when none matches, it is told so and asked to add this name to the right category's notes.
Run:
procedure_run("wechat/发朋友圈", { text }), replayed step by step without the model.Learn: when nothing fits, the agent does the task with
page_*ordesktop, then callsprocedure_saveandprocedure_note.Repair: a replay that meets a missing anchor or an unmet
checkstops there.It reports the step, what it expected and a snapshot of the page, and tells the agent to finish the task by hand now and save it as a new version under the same name.
A failed
outwardstep is always reported as "may already have been sent" and is never retried.
Empty means broken: an
extractwith no rows, a field empty in every row, or an emptyscriptvalue kept withasis reported as a failure at that step.After a redesign, a selector that finds nothing most often looks like "nothing new". Without this the agent would conclude "no new messages".
Steps that may legitimately be empty are marked
allowEmpty: true.
Fall back when the newest fails: when the newest version fails before anything outward happened, the two most recent old versions (
<name>.v<n>.json) are tried, newest first.Sites roll out redesigns gradually and run A/B tests, so a version repaired for the new page may fail on the old one.
When an old version works, its result is used, followed by a note saying which step of the newest version failed and which version ran instead.
In dsh-world's experiments on a mock site:
With empty results counted as failures, silent failures fell from 105 replays to 0.
With the fallback, repairs that broke the old page fell from 9 to 0.
Procedures that survived learning, a redesign and a repair rose from 9/25 to 16/25.
See Experiments.
Procedure files
<library>/<category>/<name>.json, with each category's notes in <category>/目录.md.
Editing by hand: the agent can also read and edit these with its own file tools; changes take effect at the next lookup or run.
Checked on save: saving validates the file and refuses it with the reason when it is wrong. An unknown key is refused too, so a typo can never quietly turn into "any link".
Kept versions: before a save overwrites a procedure, the old one is kept as
<name>.v<n>.json, the two most recent per name.Invalid files: malformed files, a file name that does not match
name, and acategorythat does not match the folder are skipped and recorded as faults, without affecting other procedures.
Field | Meaning |
| Procedure name and file name: letters (Chinese too), digits, |
| Category and folder name, same rules; defaults to |
| What it does |
|
|
| Omitted for web procedures; |
| 1–40 steps, one action each |
| A routine: run by |
| Template for the routine's result line; may use |
| Files the rows of |
Web steps ({{param}} in text is filled in):
Action | Form |
| an address |
| an anchor |
|
|
| a key name (as in |
|
|
|
|
|
|
| JS that may |
Anchors:
An anchor is
{ role?, name?, text?, css?, nth? }, with at least one of them given.roleandnameare aspage_readshows them.Procedures store neither coordinates nor snapshot refs.
click,type,pressandscriptmay takeoutward: true: this step sends something out.
Desktop steps:
The actions:
open(program name or window title),click {x,y,double?,right?,hold?},drag {x,y,x2,y2,right?,hold?},type {text,x?,y?},press(optionally withhold),scroll {direction,x?,y?}andwait {seconds}.xandyare pixels of a screenshot.For
desktop, the screenshot is of the window, so they follow the window's size, not its position.For
screen, it is of the whole screen.
A replay does not pause between steps. Wherever a person waited (a window that appears a moment after a click) the procedure needs a
wait. Saving points out where one may be missing, and a replay that stops there says so.
{
"name": "看未读",
"category": "bilibili",
"description": "看 B 站消息页的未读",
"onAlarm": true,
"summary": "私信 {{count}} 条",
"steps": [
{ "open": "https://message.bilibili.com/" },
{ "wait": { "text": "我的消息" } },
{ "extract": { "list": { "role": "listitem" }, "fields": { "who": { "role": "link" }, "text": { "role": "paragraph" } }, "limit": 8 }, "allowEmpty": true }
]
}The user's computer
Windows only. Elsewhere, turning it on records a fault and no desktop tool appears.
Nothing to install: underneath is the plugin's own win32.ps1. It takes window pictures with PrintWindow, posts mouse and keyboard messages with PostMessage, clicks tray icons through UI Automation, and starts programs from the Start menu (Get-StartApps).
Background mode (--desktop background)
The agent sees one window's screenshots, and input is posted straight to that window. Your mouse pointer never moves and the focus stays yours, so the agent can work while you use the computer. A window covered by others can still be pictured and clicked.
open finds the program in this order:
An open window. A minimized one is restored in place.
A tray icon, which it clicks.
The Start menu, which launches the program and then gives the focus back to your window.
A program that is running with neither a window nor a tray icon is not launched again, since WeChat 4.x would open a second copy.
Programs running as administrator (many games) are out of reach; the agent is told to ask you for foreground mode.
Foreground mode (--desktop foreground)
Approval: at startup Windows asks once for administrator rights. Once approved, an elevated helper runs for the rest of this session. If you decline, or do not answer within 2 minutes, the plugin stays in background mode and records the reason as a fault.
What changes: the agent sees the whole screen and uses the real mouse and keyboard, so games and administrator programs work. Your mouse moves with it, so this is for when you hand the computer over.
Limits of the elevated helper:
It only takes pictures, moves the mouse, types, and draws the banner.
Listing windows and starting programs stay with the ordinary helper, so nothing the agent opens runs as administrator.
It talks to the plugin over a pipe whose name only this session knows, after proving a one-time key.
Games: their terms usually forbid automation, and long unattended play risks a ban. That is your call.
desktop actions:
action | What it does |
|
|
| Another screenshot |
|
|
| From |
|
|
|
|
|
|
| The area from |
| Records something that moves (a video): |
| Stops a recording under way |
| Looks at the recording: no arguments for an overview sheet, |
Every action is followed by a fresh screenshot, and after a click a red circle marks where it actually landed, so the agent checks its own work. Each screenshot also says which other windows the program has open, and which appeared or closed after the last input.
You can see it act, and stop it any time
While the agent acts, a dark capsule floats at the top centre of the screen: "正在操作" (" is operating"), with "press Esc twice to stop" and a stop button. A soft blue glow surrounds the window it works in, or the whole screen in foreground mode.
To stop it, press Esc twice within half a second, or click the capsule's button:
The step under way is void. The agent is told the user pressed stop, not to retry, and to ask the user first. A recording stops where it is, and no further keys are sent to that window.
If no step was under way, the agent's next tool result carries the same message.
Until you release it, every action on the computer is refused; looking at a finished recording still works.
The stop is kept in
<data>/desktop-stopped, so restarting the agent does not lift it.When you are satisfied, run
evolution-computer-use resumein a terminal, adding--dataif the data folder is not the default.
Inbox
While desktop control is on, the plugin watches the taskbar for the agent: flashing buttons, and system notifications with a short excerpt (at most 60 characters).
When it tells: signals come in bursts. Once 10 s pass with nothing new, the burst is gathered into the inbox, and the agent's next tool result carries a one-line note that the inbox has news. The content waits for the agent to read it with
inbox_read; nothing is pushed into the middle of its work.What it reads: a program is named by its own description, never its window title, which may contain message text.
What it keeps: nothing is written to disk. Notifications already present at startup count as old.
Exclusions: programs that already reach the agent another way can be excluded with
--inbox-ignore.Permissions: if notifications cannot be read, that is recorded as a fault. Allow access in Windows Settings › Privacy & security › Notifications.
Message log
A procedure can declare with messages which rows of an extract are messages:
Fixed values:
channelandconversationType(privateorgroup).Templates: every other field is a
{{field}}template.Required:
accountId,conversationId,messageId,speakerIdandcontent.Optional:
speakerName,conversationName,occurredAtandoutgoing;outgoingevaluating totruemarks the agent's own messages.
When the agent runs such a procedure with procedure_run, what it read is appended to <data>/messages.jsonl:
Records are de-duplicated by the platform's message id, so reading again adds nothing.
Rows missing an identity field are skipped.
A failed write never fails the procedure.
Routine runs are not logged.
Embedding hosts can plug in their own MessageSink.
Multiple agents
Share what they learn: point several agents at the same
--procedures; the default data folder already is the same one. What one agent learns, the others find withcomputer_use. Changes are picked up by file time, with no restart.One browser each: a browser profile suits one agent at a time, since the queue lives in one process. Give each concurrent agent its own
--profile, and log in once in each.One agent on the desktop at a time: two agents on the same computer would interrupt each other.
Migrating from dsh-world
Procedures: the file format is identical. Copy
<world>/procedures/to<data>/procedures/; kept versions and目录.mdnotes work as before, and dsh-world's flat legacy files are still read as未分类.Logins: to keep your site logins, also copy
<world>/browser/to<data>/browser/, with that browser closed.Routines:
onAlarmprocedures ran before the alarm in dsh-world. Here they run throughprocedure_routinesor a scheduledevolution-computer-use routines.Messages: they are written to
messages.jsonlinstead ofarchive.sqlite.
Files
Location | What it is |
| The agent's browser profile, including site logins. Do not share it, sync it to cloud storage, or commit it |
| Learned procedures, kept versions and category notes; the plugin writes here only on save |
| Messages that procedures read |
| Present only while a stop is in force |
Screenshots and recording frames pass through the system temp folder and are deleted after use.
Risks
Platform rules: automating websites may trip a platform's risk controls or break its terms, and accounts may be limited or banned.
Use a dedicated secondary account in the agent's browser.
The plugin does not promise to go undetected, and there is no reliably "safe rate".
Nothing limits whom the agent talks to: it can do anything your account can, including private messages and comments. Put any limits in the agent's own prompt.
Your computer: with desktop control on, the agent sees everything in the windows it opens and can use every account signed in on this computer. Restart without
--desktopto take it back.Your browser profile: any program on this machine can read
<data>/browser/DevToolsActivePortand drive the logged-in browser. Do not run untrusted programs as the same Windows user.
Verification
npm test runs every offline test: 154, of which 2 live tests are skipped by default. ECU_EDGE_TEST=1 launches a real Edge; ECU_DESKTOP_TEST=1 takes read-only screenshots of a desktop with WeChat open. npm run typecheck checks types.
On Windows 11 Home (Chinese, 10.0.26200) with Node 22.16, the built lib/cli.js was exercised over MCP:
initialization and the tool list;
listing windows in background mode, through the real PowerShell helper;
page_openandpage_shot, in real Edge;saving and replaying a procedure;
breaking the procedure on purpose, after which it fell back to the old version.
The desktop checks done on real hardware in dsh-world use the same code but were not repeated here: background clicks, typing, bringing WeChat up from the tray, and foreground mode.
Not yet verified:
the browser side on macOS and Linux;
machines with Chrome but no Edge;
real use inside Codex, Cursor and other clients.
Experiments
How procedures are stored, replayed and repaired was settled by two controlled experiments during dsh-world's development: a real DSH container, deepseek-flash, a mock site in three redesigns, five tasks, and decision rules written before the results.
Round | Question | Outcome |
1 (2026-11-12) | Pure JSON, JSON + in-page scripts, or a whole script? | End-to-end success tied (9 / 8 / 9 of 25). Pure JSON cost far more on the paging task. By the "simpler wins a tie" rule: JSON + in-page scripts |
2 (2026-11-13) | How much do three replay algorithms help, and does saving procedures beat doing the task fresh every time? | K, empty results count as failures: silent failures 105 → 0. V, fall back to kept versions: end-to-end success 9/25 → 16/25. F, fingerprint self-heal: uncertain. Doing it fresh was more reliable but cost about 4× as much |
This plugin implements K and V. F, and a save-time check for class selectors (which the reports name as the most promising next step), are not implemented.
The full write-ups and the mapping to this code are in docs/experiments/. The original Chinese reports and pre-registered designs are in docs/experiments/zh-CN/.
Credits
The core is ported from dsh-world 0.8, src/ecosystem/computer/ (MIT). The browser discovery and launch, the minimal CDP session and the in-page element registry were ported from dsh-inner-browser-use 0.47.28 (MIT) by way of dsh-world, with attribution and license kept; see NOTICE.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
Let AI agents use your real, signed-in Chrome: tabs, clicks, forms, screenshots and video.
Let your AI agent operate any website through a real, logged-in browser session.
Give AI agents a real cloud Chrome: open sessions, navigate, read pages, click, type and screenshot.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables MCP-speaking AI agents to operate a desktop and browser through structured UI inspection, mouse/keyboard gestures, tab-scoped browsing, and approved local screen recording with editing.31MIT
- AlicenseBqualityBmaintenanceEnables coding agents to operate desktop apps or VMs, capture successful GUI runs as local workflow memories, and replay reusable tasks with visual checks.172MIT
- AlicenseAqualityBmaintenanceEnables agents to perceive and control a Windows desktop, a connected local Chromium browser, or a cloud Playwright browser through human-like mouse moves, clicks, drags, scrolling, and screenshots, forming an observe-locate-act computer-use loop. Includes safety guards such as random pauses, emergency stop, and approval prompts for destructive actions.11MIT

RecordForgeofficial
AlicenseNot gradedqualityBmaintenanceEnables agents to capture real browser interactions over CDP, compile them into deterministic replay scripts with parameterized inputs and stored secrets, and execute those workflows with real input events — no page re-reading or LLM in the replay loop. Also emits portable Agent Skills from recorded workflows.38 npm1AGPL 3.0