agent-desktop
Provides guarded control of a Linux desktop, enabling AI agents to perform mouse, keyboard, window, clipboard, and screenshot actions through a policy engine with optional human confirmation, secret redaction, and audit logging.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
agent-desktop — guarded desktop & browser control MCP
An MCP server that lets an AI agent drive a Linux desktop and browser — behind a policy engine. Every action passes through rules, optional human confirmation, secret redaction, and an append-only audit trail.
The premise: agents that click, type, and screenshot need guardrails more than they need more capabilities. This server makes the guardrails the architecture.
Architecture
agent/desktop/
├── server.py # MCP server — 37 tools, wiring only
├── policy.py # PolicyEngine: rules, verdicts (allow/confirm/deny), confirm tokens
├── confirm.py # ConfirmStore: TTL'd human-approval tokens
├── audit.py # AuditLog: append-only JSONL record of every action
├── redact.py # Secret redaction before anything leaves the machine
├── browser.py # Playwright controller with commit/rollback scopes
├── desktop.py # X11 desktop control (mouse, keyboard, windows, clipboard)
├── openrouter.py # Optional vision via OpenRouter (API key stays in env)
├── session.py # Desktop session store — gating per session
└── tests/ # pytest suite; policy tests run without a browser or X serverRequest flow: tool call → PolicyEngine verdict → (confirm token if required) → executor → redaction → audit entry.
Related MCP server: mcp-linux-desktop
Tool families
Browser (16) —
browser_navigate,browser_click,browser_fill,browser_snapshot,browser_screenshot,browser_tabs,browser_enter_scope/browser_exit_scope(commit/rollback),browser_visual_query, …Desktop (16) —
desktop_mouse_move,desktop_mouse_click,desktop_type_text,desktop_key,desktop_window_list,desktop_clipboard_get/_set,desktop_screenshot,desktop_visual_query, …Governance —
policy_status,policy_reload,desktop_confirm(approve a pending action),desktop_session_open/_close/_status,vision_status
Install
git clone <this repo> && cd agent-desktop
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/playwright install chromium # for browser tools
pytest agent/desktop/tests # policy tests, no X server neededRun: python -m agent.desktop.server from the repo root.
Policy configuration
Rules load from a JSON policy file (default
~/local-agent/config/desktop-policy.json, overridable). The policy defines
which actions are allowed, which require an explicit confirm token, protected
paths, and the audit log location. API keys are read from environment
variables — never from the policy file.
How I built this
The problem. A desktop agent that can click anything is a liability, not a feature. The interesting part of this project is not the browser automation — it is deciding, in code, which actions are allowed to happen without asking.
Design decisions worth arguing about:
The policy engine is the product, not a wrapper. Every tool call passes through
PolicyEnginebefore it executes. Tool implementations do not decide their own permissions, so a new tool cannot accidentally become an unguarded one.Confirmation tokens, not UI prompts. A prompt can be dismissed; a token cannot be forged. The caller receives a token bound to a specific action and must present it back. Friction applied everywhere is friction people route around, so confirmation is scoped to the destructive path only.
Redaction happens at the tool boundary. Secrets are stripped from tool output before it reaches the model, not before it reaches the user. The model never sees a credential it could echo into a transcript.
JSONL, not a database. An append-only log is greppable, cheap, and impossible to lose in a way you don't notice. It also survives the agent crashing, which is exactly when you want the record.
Tests over demos. 41 tests cover the policy paths — commit, session, confirmation, browser — because the failure mode of this project is a policy that silently permits too much.
What I would change with more time: the policy rules are currently code rather than configuration, which means changing them requires a deploy. A declarative policy file with a schema would make the safe thing the easy thing for whoever operates it.
License
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Persistent Linux computers for AI agents: desktop, signed-in browser, vault, apps and shell.
Related MCP Servers
- AlicenseBqualityDmaintenanceProvides AI assistants with the ability to control Linux desktop environments through tools for file management, application launching, and system operations like clipboard access. It includes a multi-level security model to manage permissions for safe, elevated, and restricted actions.6MIT
- AlicenseNot gradedqualityDmaintenanceEnables full Linux desktop control including windows, mouse, keyboard, clipboard, audio, screenshots, OCR, accessibility, and system management through MCP-compatible AI agents.1MIT
- AlicenseNot gradedqualityCmaintenanceEnables an AI agent to see and control a Linux desktop via Wayland/Hyprland, providing screenshots, structured desktop state, pointer/keyboard input, semantic window/workspace tools, and a policy engine for safe execution.Apache 2.0
- AlicenseNot gradedqualityBmaintenanceEnables tool-capable AI models to operate a private Linux desktop and headless browser, with terminal, file, and application controls, while keeping your physical mouse and keyboard separate.MIT