NexcoreCtrlComputer
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Your agent does not need a 4K screenshot to click a button called Save.
NexcoreCtrlComputer is the open-source control kernel behind NexcoreWork. It exposes one JS/CLI/MCP contract for desktop and browser control, resolves accessibility targets locally, verifies important postconditions, and returns a compact receipt instead of dumping screenshots and raw UI trees into the model.
The durable unit can be a one-off semantic action or a parameterized UI program. The kernel is built on agent-desktop, agent-browser, and an explicitly gated PyAutoGUI fallback. It is not a standalone desktop app.
What changed in 0.4.1
0.4.1 adds installed-software discovery and an adapter registry on top of the
0.4.0 layered Windows fallback:
observe({ view: "software" })lists launchable Start Menu applications with their exact launch names and AUMIDs.observe({ view: "adapters" })reports desktop, browser, vision, workflow, and local CLI adapters without pretending optional engines are installed.Software results include
launchAdapterandcontrolAdapters, so the model can open the exact app and then choose the right semantic control route.Windows-MCPcan launch applications, capture desktop/window state, resolve labels, synthesize input, and wait for text when the primary desktop backend is unavailable.Desktop fallback remains behind the same CLI/JS/MCP contract and returns the same route/delivery/verification evidence.
The browser-to-desktop routing fix from
0.3.0remains enforced.
0.3.0 fixed the failure mode where an open browser session hijacked later
desktop work:
observenow supports desktop inventory views:windows,apps,displays,notifications, andsurfaces.Inventory views always return to the desktop route, even after a browser task.
Window focus restores minimized windows and retries transient foreground-lock failures.
Window management and clipboard actions are part of the same action contract.
property=textreads actual accessible text instead of treating existence as a text match.Every action can return route, delivery, retry, and verification evidence.
The Windows live test used in this repository opens the real Calculator,
observes its accessible controls, clicks the 七 button, and verifies the text
显示为 7.
route: desktop / windows / provided-ref
action: click button "七"
proof: text = "显示为 7"
result: delivered_verifiedRelated MCP server: mcp-computer-use
Why this exists
Problem | Nexcore's answer |
The same workflow is replanned every run | Parameterized semantic programs |
Desktop and browser plans live only in prompts | One cross-surface executable artifact |
Screenshots are expensive and ambiguous | Semantic role/name/ref snapshots |
Coordinates break when DPI or layout changes | Accessibility-first actions |
Dozens of tools bloat agent context | One |
| Optional post-action |
Dense apps overflow context | Node caps, exact queries, drill-down refs |
Blind retries double-click real controls | Backend retry disposition is preserved |
Pixel automation is unsafe by default | Headed and coordinate modes are opt-in |
Every agent reimplements backend quirks | One normalized JS/CLI/MCP contract |
The new primitive
Semantic programs store intent and contracts, never screenshots, coordinates, or snapshot-qualified refs:
{
"schema": "nexcore.program/v1",
"name": "browser.fill",
"version": "1.0.0",
"params": {
"value": { "type": "string", "required": true }
},
"steps": [
{
"act": {
"surface": "browser",
"action": "fill",
"target": { "role": "textbox", "name": "Name" },
"text": "{{value}}",
"expect": { "property": "value", "value": "{{value}}" }
}
}
]
}nexcore program save ./examples/programs/browser-fill.json
nexcore program run browser.fill '{"value":"Ada"}'The model does not resend the steps. It sends one call:
{
"mode": "program_run",
"programName": "browser.fill",
"params": { "value": "Ada" }
}At replay time:
parameters are type-checked and interpolated;
desktop targets try exact accessibility find, then named-context scoping, then compact snapshots;
browser selectors and semantic targets are resolved inside the active session;
every declared postcondition is checked;
unsafe post-delivery retries are refused.
Read the program reference for the complete schema.
Black-box prompts and expected outcomes are in docs/TEST-CASES.zh-CN.md. The latest five-round Windows UI test report is in docs/LIVE-TEST-REPORT-2026-11-15.md.
Architecture
The surface router keeps desktop and browser state explicit. The local runtime
owns bounded observation, target resolution, policy, action execution, and state
proof. A previous browser session cannot silently capture a later
observe({ view: "windows" }) call.
Which observation should I use?
Goal | Call |
Find an open desktop window |
|
List running desktop applications |
|
Inspect Calculator controls |
|
Continue working in the active page |
|
Open a URL and inspect it |
|
30-second demo
git clone https://github.com/yan-stone-computer/NexcoreCtrlComputer.git
cd NexcoreCtrlComputer
npm install
npm link
npm install -g agent-desktop agent-browser
python -m pip install pyautogui # optional fallback
nexcore doctor
nexcore observe '{"view":"windows"}'
nexcore observe '{"view":"apps"}'
nexcore observe '{"surface":"desktop","app":"Calculator","query":"7"}'
nexcore act '{"surface":"desktop","app":"Calculator","action":"click","target":{"role":"button","name":"7"}}'Desktop refs look like this:
desktop app "Calculator"
@sabc123:e1 textfield "Display" = "0" {enabled} [setvalue,focus]
@sabc123:e2 button "7" [click]A browser run looks like this:
nexcore act '{
"surface": "browser",
"url": "https://example.com",
"action": "click",
"target": {"role":"link","name":"Learn more"},
"expect": {
"property": "url",
"value": "https://www.iana.org/help/example-domains"
}
}'The second step opens the page, finds the exact semantic target, clicks it, and verifies navigation. No screenshot is sent to the model.
One tool, not fifty
nexcore mcp exposes a single computer tool over stdio:
{
"mcpServers": {
"nexcore": {
"command": "nexcore",
"args": ["mcp"],
"env": {
"NEXCORE_SESSION": "default",
"NEXCORE_ALLOWED_APPS": "Calculator,notepad.exe",
"NEXCORE_ALLOWED_DOMAINS": "example.com"
}
}
}
}Modes: status, observe, act, run, program_save, program_list,
program_run, close.
Actions: open, click, double-click, right-click, hover, focus, fill, type,
press, select, check, uncheck, toggle, expand, collapse, scroll, window
management, and clipboard read/write. Desktop inventory is available through
observe views for windows, apps, displays, notifications, and surfaces. These
views always return to the desktop route, even after a browser session.
Coordinate click and screenshot remain explicit one-off actions and are rejected
inside stored programs.
This keeps the tool catalog stable while the agent performs multi-step work:
{
"mode": "run",
"steps": [
{
"observe": {
"surface": "desktop",
"app": "Calculator",
"query": "7"
}
},
{
"act": {
"surface": "desktop",
"app": "Calculator",
"action": "click",
"target": {
"role": "button",
"name": "7"
}
}
}
]
}Use it from JavaScript
Link the local open-source package into your project:
cd NexcoreCtrlComputer
npm link
cd ../your-agent-project
npm link nexcorectrlcomputerimport { createComputer } from "nexcorectrlcomputer";
const computer = await createComputer({
session: "checkout-audit",
policy: {
allowedApps: ["Calculator"],
allowedDomains: ["example.com"],
allowHeaded: false,
allowCoordinates: false
}
});
const view = await computer.observe({
surface: "browser",
url: "https://example.com"
});
console.log(view.text);
console.log(view.estimatedTokens);
await computer.act({
surface: "browser",
action: "click",
target: { role: "link", name: "Learn more" }
});
await computer.verify({
surface: "browser",
property: "url",
value: "https://www.iana.org/help/example-domains"
});
await computer.close();The same API can be mounted inside OpenAI Responses function calling, LangGraph, AutoGen, CrewAI, Claude, Codex, or a custom MCP client.
Token reduction without hand-waving
The included fixture benchmark compares raw accessibility JSON with a compact observation, then compares an inline plan with a program replay call:
npm run bench30-node synthetic accessibility tree
raw: 8,406 bytes (~2,102 estimated tokens)
compact: 2,155 bytes (~539 estimated tokens)
reduction: 74.4%
two-step browser plan
inline: 213 bytes (~54 estimated tokens)
program: 85 bytes (~22 estimated tokens)
reduction: 60.1%These are UTF-8 byte and bytes / 4 estimates on a synthetic fixture, not a
provider bill. Real savings depend on tree density and how narrowly you query.
The larger win is architectural: screenshots stay local unless visual grounding
is genuinely necessary.
Observe -> act -> verify
Observe the smallest useful region.
Act on a fresh accessibility ref or exact semantic target.
Verify a value, state, or URL when the action matters.
Retry only when the backend says retry is safe.
const result = await computer.act({
surface: "desktop",
app: "Calculator",
action: "click",
target: "@sabc123:e2",
expect: {
property: "value",
value: "7",
timeoutMs: 3000
}
});Actions are not blindly replayed. A click may already have landed; backend disposition prevents duplicate input.
Safety model
Desktop and browser actions use semantic APIs before physical input.
Headed input is denied unless
allowHeadedis true.Coordinates are denied unless
allowCoordinatesis true.App and domain allowlists are supported.
PyAutoGUI keeps its corner fail-safe enabled.
Backend output is treated as untrusted content.
nexcore closecloses the browser session, not the user's applications.Secrets should stay out of prompts, traces, screenshots, and process args.
See SECURITY.md before unattended use.
Install backends
Nexcore has zero npm runtime dependencies. The specialist engines are optional, separately versioned backends:
npm install agent-desktop agent-browser
python -m pip install -r requirements-coordinate.txtEnvironment overrides:
NEXCORE_DESKTOP_BIN absolute backend path
NEXCORE_BROWSER_BIN absolute backend path
NEXCORE_PYTHON_BIN Python executable
NEXCORE_SESSION isolated session name
NEXCORE_PROGRAM_DIR semantic program registry directory
NEXCORE_ALLOW_HEADED "1" to enable physical input
NEXCORE_ALLOW_COORDINATES "1" to enable PyAutoGUI coordinate actions
NEXCORE_ALLOWED_APPS comma-separated desktop allowlist
NEXCORE_ALLOWED_DOMAINS comma-separated browser allowlistStatus
Validated on Windows with Node 24, agent-desktop 1.2.0,
agent-browser 0.42.0, and PyAutoGUI 0.9.54:
accessibility grant and backend doctor
live desktop snapshot and value verification
live browser click and URL verification
live parameterized browser program save and verified replay
live cross-surface program spanning a Chromium page and a native app
browser-to-desktop inventory switching without session leakage
real Calculator launch, semantic button click, and accessible-text verification
MCP protocol smoke test
unit tests for compiler, storage, contextual resolution, and policies
macOS and Linux support comes through the same upstream accessibility engines, but is not yet covered by this repository's live CI. Contributions that run real platform checks are especially valuable.
Related projects
NexcoreCtrlComputer is independent and does not vendor upstream source:
agent-desktop - native accessibility snapshots and actions, Apache-2.0
agent-browser - browser automation and CDP snapshots, Apache-2.0
PyAutoGUI - coordinate input and screenshot fallback, BSD-3-Clause
See NOTICE for the integration boundary.
Contributing
The best contribution is a smaller round trip, a safer failure, or a real
platform verification. Read CONTRIBUTING.md, run npm test,
and include evidence from a live backend when changing adapter arguments.
If this saves your agent from one screenshot loop, star the repo and bring your hardest computer-use workflow.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Use a person's computer after they approve it there: files, Terminal, browser, local MCP.
Use a person's computer after they approve it there: files, Terminal, browser, local MCP.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
Related MCP Servers
- AlicenseAqualityAmaintenanceThe local MCP server that gives any AI agent safe desktop control. Provides 6 compact tools (computer, accessibility, window, system, browser, task) for cross-platform GUI automation with ground-truth verification.261,371 npm406MIT
- AlicenseAqualityCmaintenanceEnables MCP-speaking clients to control a real desktop via the computer_use tool, including clicking, typing, scrolling, dragging, key combos, app focus, and screen/accessibility capture. It also provides a verdict system that verifies whether input actions had their intended effect, with configurable approval modes.1MIT
- AlicenseAqualityCmaintenanceEnables MCP-speaking AI agents to operate a desktop and browser through structured UI inspection, mouse/keyboard gestures, tab-scoped browsing, and approved local screen recording with editing.31MIT
- AlicenseNot gradedqualityCmaintenanceEnables an MCP host to observe and control Windows desktop applications deterministically: reading UIA control trees, OCR text and pixel diffs, clicking, typing, scrolling and window management, plus verbatim text reading/appending via ValuePattern/TextPattern. Visual judgment is delegated back to the host model's own multimodal context through annotated screenshots, numbered element cards, grid maps and crosshair zoom-ins, so no separate vision model is needed.1AGPL 3.0