Skip to main content
Glama

Your agent does not need a 4K screenshot to click a button called Save.

NexcoreCtrlComputer is the open-source control kernel behind NexcoreWork. It exposes one JS/CLI/MCP contract for desktop and browser control, resolves accessibility targets locally, verifies important postconditions, and returns a compact receipt instead of dumping screenshots and raw UI trees into the model.

The durable unit can be a one-off semantic action or a parameterized UI program. The kernel is built on agent-desktop, agent-browser, and an explicitly gated PyAutoGUI fallback. It is not a standalone desktop app.

What changed in 0.4.1

0.4.1 adds installed-software discovery and an adapter registry on top of the 0.4.0 layered Windows fallback:

  • observe({ view: "software" }) lists launchable Start Menu applications with their exact launch names and AUMIDs.

  • observe({ view: "adapters" }) reports desktop, browser, vision, workflow, and local CLI adapters without pretending optional engines are installed.

  • Software results include launchAdapter and controlAdapters, so the model can open the exact app and then choose the right semantic control route.

  • Windows-MCP can launch applications, capture desktop/window state, resolve labels, synthesize input, and wait for text when the primary desktop backend is unavailable.

  • Desktop fallback remains behind the same CLI/JS/MCP contract and returns the same route/delivery/verification evidence.

  • The browser-to-desktop routing fix from 0.3.0 remains enforced.

0.3.0 fixed the failure mode where an open browser session hijacked later desktop work:

  • observe now supports desktop inventory views: windows, apps, displays, notifications, and surfaces.

  • Inventory views always return to the desktop route, even after a browser task.

  • Window focus restores minimized windows and retries transient foreground-lock failures.

  • Window management and clipboard actions are part of the same action contract.

  • property=text reads actual accessible text instead of treating existence as a text match.

  • Every action can return route, delivery, retry, and verification evidence.

The Windows live test used in this repository opens the real Calculator, observes its accessible controls, clicks the 七 button, and verifies the text 显示为 7.

route:  desktop / windows / provided-ref
action: click button "七"
proof:  text = "显示为 7"
result: delivered_verified

Related MCP server: mcp-computer-use

Why this exists

Problem

Nexcore's answer

The same workflow is replanned every run

Parameterized semantic programs

Desktop and browser plans live only in prompts

One cross-surface executable artifact

Screenshots are expensive and ambiguous

Semantic role/name/ref snapshots

Coordinates break when DPI or layout changes

Accessibility-first actions

Dozens of tools bloat agent context

One computer MCP tool

ok: true does not prove the UI changed

Optional post-action expect

Dense apps overflow context

Node caps, exact queries, drill-down refs

Blind retries double-click real controls

Backend retry disposition is preserved

Pixel automation is unsafe by default

Headed and coordinate modes are opt-in

Every agent reimplements backend quirks

One normalized JS/CLI/MCP contract

The new primitive

Semantic programs store intent and contracts, never screenshots, coordinates, or snapshot-qualified refs:

{
  "schema": "nexcore.program/v1",
  "name": "browser.fill",
  "version": "1.0.0",
  "params": {
    "value": { "type": "string", "required": true }
  },
  "steps": [
    {
      "act": {
        "surface": "browser",
        "action": "fill",
        "target": { "role": "textbox", "name": "Name" },
        "text": "{{value}}",
        "expect": { "property": "value", "value": "{{value}}" }
      }
    }
  ]
}
nexcore program save ./examples/programs/browser-fill.json
nexcore program run browser.fill '{"value":"Ada"}'

The model does not resend the steps. It sends one call:

{
  "mode": "program_run",
  "programName": "browser.fill",
  "params": { "value": "Ada" }
}

At replay time:

  1. parameters are type-checked and interpolated;

  2. desktop targets try exact accessibility find, then named-context scoping, then compact snapshots;

  3. browser selectors and semantic targets are resolved inside the active session;

  4. every declared postcondition is checked;

  5. unsafe post-delivery retries are refused.

Read the program reference for the complete schema.

Black-box prompts and expected outcomes are in docs/TEST-CASES.zh-CN.md. The latest five-round Windows UI test report is in docs/LIVE-TEST-REPORT-2026-11-15.md.

Architecture

NexcoreCtrlComputer desktop and browser control architecture

The surface router keeps desktop and browser state explicit. The local runtime owns bounded observation, target resolution, policy, action execution, and state proof. A previous browser session cannot silently capture a later observe({ view: "windows" }) call.

Which observation should I use?

Goal

Call

Find an open desktop window

observe({ view: "windows" })

List running desktop applications

observe({ view: "apps" })

Inspect Calculator controls

observe({ app: "CalculatorApp.exe", query: "7" })

Continue working in the active page

observe({ surface: "browser" })

Open a URL and inspect it

observe({ surface: "browser", url: "https://..." })

30-second demo

git clone https://github.com/yan-stone-computer/NexcoreCtrlComputer.git
cd NexcoreCtrlComputer
npm install
npm link
npm install -g agent-desktop agent-browser
python -m pip install pyautogui                          # optional fallback
nexcore doctor

nexcore observe '{"view":"windows"}'
nexcore observe '{"view":"apps"}'
nexcore observe '{"surface":"desktop","app":"Calculator","query":"7"}'
nexcore act '{"surface":"desktop","app":"Calculator","action":"click","target":{"role":"button","name":"7"}}'

Desktop refs look like this:

desktop app "Calculator"
@sabc123:e1 textfield "Display" = "0" {enabled} [setvalue,focus]
@sabc123:e2 button "7" [click]

A browser run looks like this:

nexcore act '{
  "surface": "browser",
  "url": "https://example.com",
  "action": "click",
  "target": {"role":"link","name":"Learn more"},
  "expect": {
    "property": "url",
    "value": "https://www.iana.org/help/example-domains"
  }
}'

The second step opens the page, finds the exact semantic target, clicks it, and verifies navigation. No screenshot is sent to the model.

One tool, not fifty

nexcore mcp exposes a single computer tool over stdio:

{
  "mcpServers": {
    "nexcore": {
      "command": "nexcore",
      "args": ["mcp"],
      "env": {
        "NEXCORE_SESSION": "default",
        "NEXCORE_ALLOWED_APPS": "Calculator,notepad.exe",
        "NEXCORE_ALLOWED_DOMAINS": "example.com"
      }
    }
  }
}

Modes: status, observe, act, run, program_save, program_list, program_run, close.

Actions: open, click, double-click, right-click, hover, focus, fill, type, press, select, check, uncheck, toggle, expand, collapse, scroll, window management, and clipboard read/write. Desktop inventory is available through observe views for windows, apps, displays, notifications, and surfaces. These views always return to the desktop route, even after a browser session. Coordinate click and screenshot remain explicit one-off actions and are rejected inside stored programs.

This keeps the tool catalog stable while the agent performs multi-step work:

{
  "mode": "run",
  "steps": [
    {
      "observe": {
        "surface": "desktop",
        "app": "Calculator",
        "query": "7"
      }
    },
    {
      "act": {
        "surface": "desktop",
        "app": "Calculator",
        "action": "click",
        "target": {
          "role": "button",
          "name": "7"
        }
      }
    }
  ]
}

Use it from JavaScript

Link the local open-source package into your project:

cd NexcoreCtrlComputer
npm link
cd ../your-agent-project
npm link nexcorectrlcomputer
import { createComputer } from "nexcorectrlcomputer";

const computer = await createComputer({
  session: "checkout-audit",
  policy: {
    allowedApps: ["Calculator"],
    allowedDomains: ["example.com"],
    allowHeaded: false,
    allowCoordinates: false
  }
});

const view = await computer.observe({
  surface: "browser",
  url: "https://example.com"
});

console.log(view.text);
console.log(view.estimatedTokens);

await computer.act({
  surface: "browser",
  action: "click",
  target: { role: "link", name: "Learn more" }
});

await computer.verify({
  surface: "browser",
  property: "url",
  value: "https://www.iana.org/help/example-domains"
});

await computer.close();

The same API can be mounted inside OpenAI Responses function calling, LangGraph, AutoGen, CrewAI, Claude, Codex, or a custom MCP client.

Token reduction without hand-waving

The included fixture benchmark compares raw accessibility JSON with a compact observation, then compares an inline plan with a program replay call:

npm run bench
30-node synthetic accessibility tree
raw:     8,406 bytes  (~2,102 estimated tokens)
compact: 2,155 bytes    (~539 estimated tokens)
reduction: 74.4%

two-step browser plan
inline:     213 bytes  (~54 estimated tokens)
program:     85 bytes  (~22 estimated tokens)
reduction: 60.1%

These are UTF-8 byte and bytes / 4 estimates on a synthetic fixture, not a provider bill. Real savings depend on tree density and how narrowly you query. The larger win is architectural: screenshots stay local unless visual grounding is genuinely necessary.

Observe -> act -> verify

  1. Observe the smallest useful region.

  2. Act on a fresh accessibility ref or exact semantic target.

  3. Verify a value, state, or URL when the action matters.

  4. Retry only when the backend says retry is safe.

const result = await computer.act({
  surface: "desktop",
  app: "Calculator",
  action: "click",
  target: "@sabc123:e2",
  expect: {
    property: "value",
    value: "7",
    timeoutMs: 3000
  }
});

Actions are not blindly replayed. A click may already have landed; backend disposition prevents duplicate input.

Safety model

  • Desktop and browser actions use semantic APIs before physical input.

  • Headed input is denied unless allowHeaded is true.

  • Coordinates are denied unless allowCoordinates is true.

  • App and domain allowlists are supported.

  • PyAutoGUI keeps its corner fail-safe enabled.

  • Backend output is treated as untrusted content.

  • nexcore close closes the browser session, not the user's applications.

  • Secrets should stay out of prompts, traces, screenshots, and process args.

See SECURITY.md before unattended use.

Install backends

Nexcore has zero npm runtime dependencies. The specialist engines are optional, separately versioned backends:

npm install agent-desktop agent-browser
python -m pip install -r requirements-coordinate.txt

Environment overrides:

NEXCORE_DESKTOP_BIN       absolute backend path
NEXCORE_BROWSER_BIN       absolute backend path
NEXCORE_PYTHON_BIN        Python executable
NEXCORE_SESSION           isolated session name
NEXCORE_PROGRAM_DIR       semantic program registry directory
NEXCORE_ALLOW_HEADED      "1" to enable physical input
NEXCORE_ALLOW_COORDINATES "1" to enable PyAutoGUI coordinate actions
NEXCORE_ALLOWED_APPS      comma-separated desktop allowlist
NEXCORE_ALLOWED_DOMAINS   comma-separated browser allowlist

Status

Validated on Windows with Node 24, agent-desktop 1.2.0, agent-browser 0.42.0, and PyAutoGUI 0.9.54:

  • accessibility grant and backend doctor

  • live desktop snapshot and value verification

  • live browser click and URL verification

  • live parameterized browser program save and verified replay

  • live cross-surface program spanning a Chromium page and a native app

  • browser-to-desktop inventory switching without session leakage

  • real Calculator launch, semantic button click, and accessible-text verification

  • MCP protocol smoke test

  • unit tests for compiler, storage, contextual resolution, and policies

macOS and Linux support comes through the same upstream accessibility engines, but is not yet covered by this repository's live CI. Contributions that run real platform checks are especially valuable.

NexcoreCtrlComputer is independent and does not vendor upstream source:

  • agent-desktop - native accessibility snapshots and actions, Apache-2.0

  • agent-browser - browser automation and CDP snapshots, Apache-2.0

  • PyAutoGUI - coordinate input and screenshot fallback, BSD-3-Clause

See NOTICE for the integration boundary.

Contributing

The best contribution is a smaller round trip, a safer failure, or a real platform verification. Read CONTRIBUTING.md, run npm test, and include evidence from a live backend when changing adapter arguments.

If this saves your agent from one screenshot loop, star the repo and bring your hardest computer-use workflow.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      A
      maintenance
      The local MCP server that gives any AI agent safe desktop control. Provides 6 compact tools (computer, accessibility, window, system, browser, task) for cross-platform GUI automation with ground-truth verification.
      2
      6
      1,371 npm
      406
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables MCP-speaking clients to control a real desktop via the computer_use tool, including clicking, typing, scrolling, dragging, key combos, app focus, and screen/accessibility capture. It also provides a verdict system that verifies whether input actions had their intended effect, with configurable approval modes.
      1
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables MCP-speaking AI agents to operate a desktop and browser through structured UI inspection, mouse/keyboard gestures, tab-scoped browsing, and approved local screen recording with editing.
      3
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables an MCP host to observe and control Windows desktop applications deterministically: reading UIA control trees, OCR text and pixel diffs, clicking, typing, scrolling and window management, plus verbatim text reading/appending via ValuePattern/TextPattern. Visual judgment is delegated back to the host model's own multimodal context through annotated screenshots, numbered element cards, grid maps and crosshair zoom-ins, so no separate vision model is needed.
      1
      AGPL 3.0