Poltr

English | 简体中文

Why · How it works · Quickstart · MCP tools · Support · Roadmap

Poltr is a local MCP server that gives any agent (Claude, Gemini, or your own) an isolated desktop, a demonstration recorder, and a skill engine.

🧠 The agent reasons. 🤖 Poltr stays deterministic: recording, validation, execution, verification and safety.

💡 Why

Computer-use agents re-reason every step of every run. That is slow, expensive and flaky.

Related MCP server: windows-gui-mcp

🔄 How it works

🚀 Quickstart

Requires Node 18+. The sandbox needs Linux with Xvfb, Openbox, xdotool, dbus-daemon and at-spi2-core (at-spi-bus-launcher). GTK/ATK applications such as Mousepad must expose accessibility through their toolkit bridge.

git clone https://github.com/janghotan/Poltr && cd Poltr
npm install
npm run build

node dist/src/cli/index.js sandbox start
node dist/src/cli/index.js computer probe --computer sandbox

🔗 Connect an agent over stdio

{
  "mcpServers": {
    "poltr": {
      "command": "node",
      "args": ["dist/src/cli/index.js", "mcp"]
    }
  }
}

▶️ Run a skill from the CLI

node dist/src/cli/index.js skill list
node dist/src/cli/index.js run <skill-name> --dry-run
node dist/src/cli/index.js run <skill-name> --param name=value

💡 No display handy? Add --computer mock to try everything without a desktop.

mcp · sandbox start|status|stop|reset · computer probe|info · skill list|show · run · recording list|show|delete · healing list|show

🧰 MCP tools

The agent learns progressively: list recordings → show one → fetch steps → fetch individual screenshots → submit a skill.

🧬 Skill format

A skill has typed parameters (string number boolean path command application, substituted as {{name}} at runtime), preconditions, steps and postconditions. Each step carries an intent, application, semantic target, action, expected result, verification and evidence references. See skills-library/.

📊 Current support

♿ The X11 sandbox now exposes a real AT-SPI2 accessibility tree through a sandbox-private D-Bus session. ui-element and text targets are resolved from current application/window/role/name/text/state data. When a target is resolved, its current screen bounds are used for the computer action; this is not coordinate replay. If dbus-daemon or at-spi-bus-launcher is unavailable, Poltr returns an explicit unsupported-capability result and does not inspect the host accessibility bus.

AT-SPI support is toolkit-dependent. GTK/ATK applications such as Mousepad are the primary tested case; other Linux toolkits may expose only part of their accessibility tree or require their own accessibility configuration. LocalComputer remains unchanged and does not silently fall back to the sandbox tree.

poltr skill compile is a deterministic prototype. Real skill authoring happens through the agent and poltr_save_skill.

🗺️ Roadmap

• Isolated Xvfb + Openbox desktop sandbox

• Demonstration recorder with screenshot evidence

• Skill schema, evidence provenance validation, immutable versions

• Semantic executor with deterministic verification

• Evidence-driven healing with sandboxed repair tests

• AT-SPI accessibility tree for the sandbox

• Visual and terminal-output verification

• Runnable example skills

• Reference agent that learns and runs skills end to end

• Demo recording

🛠️ Development

npm run build
npm test
npm run lint
POLTR_RUN_SANDBOX_TEST=1 npm test   # opt-in real X11 sandbox test

📐 Architecture details: docs/architecture.md

MIT © Jangho Tan · Built for agents that should not have to relearn the desktop every time