Skip to main content
Glama

agent-desktop — guarded desktop & browser control MCP

An MCP server that lets an AI agent drive a Linux desktop and browser — behind a policy engine. Every action passes through rules, optional human confirmation, secret redaction, and an append-only audit trail.

The premise: agents that click, type, and screenshot need guardrails more than they need more capabilities. This server makes the guardrails the architecture.

Architecture

agent/desktop/
├── server.py      # MCP server — 37 tools, wiring only
├── policy.py      # PolicyEngine: rules, verdicts (allow/confirm/deny), confirm tokens
├── confirm.py     # ConfirmStore: TTL'd human-approval tokens
├── audit.py       # AuditLog: append-only JSONL record of every action
├── redact.py      # Secret redaction before anything leaves the machine
├── browser.py     # Playwright controller with commit/rollback scopes
├── desktop.py     # X11 desktop control (mouse, keyboard, windows, clipboard)
├── openrouter.py  # Optional vision via OpenRouter (API key stays in env)
├── session.py     # Desktop session store — gating per session
└── tests/         # pytest suite; policy tests run without a browser or X server

Request flow: tool call → PolicyEngine verdict → (confirm token if required) → executor → redaction → audit entry.

Related MCP server: mcp-linux-desktop

Tool families

  • Browser (16) — browser_navigate, browser_click, browser_fill, browser_snapshot, browser_screenshot, browser_tabs, browser_enter_scope/browser_exit_scope (commit/rollback), browser_visual_query, …

  • Desktop (16) — desktop_mouse_move, desktop_mouse_click, desktop_type_text, desktop_key, desktop_window_list, desktop_clipboard_get/_set, desktop_screenshot, desktop_visual_query, …

  • Governance — policy_status, policy_reload, desktop_confirm (approve a pending action), desktop_session_open/_close/_status, vision_status

Install

git clone <this repo> && cd agent-desktop
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/playwright install chromium   # for browser tools
pytest agent/desktop/tests              # policy tests, no X server needed

Run: python -m agent.desktop.server from the repo root.

Policy configuration

Rules load from a JSON policy file (default ~/local-agent/config/desktop-policy.json, overridable). The policy defines which actions are allowed, which require an explicit confirm token, protected paths, and the audit log location. API keys are read from environment variables — never from the policy file.

How I built this

The problem. A desktop agent that can click anything is a liability, not a feature. The interesting part of this project is not the browser automation — it is deciding, in code, which actions are allowed to happen without asking.

Design decisions worth arguing about:

  • The policy engine is the product, not a wrapper. Every tool call passes through PolicyEngine before it executes. Tool implementations do not decide their own permissions, so a new tool cannot accidentally become an unguarded one.

  • Confirmation tokens, not UI prompts. A prompt can be dismissed; a token cannot be forged. The caller receives a token bound to a specific action and must present it back. Friction applied everywhere is friction people route around, so confirmation is scoped to the destructive path only.

  • Redaction happens at the tool boundary. Secrets are stripped from tool output before it reaches the model, not before it reaches the user. The model never sees a credential it could echo into a transcript.

  • JSONL, not a database. An append-only log is greppable, cheap, and impossible to lose in a way you don't notice. It also survives the agent crashing, which is exactly when you want the record.

  • Tests over demos. 41 tests cover the policy paths — commit, session, confirmation, browser — because the failure mode of this project is a policy that silently permits too much.

What I would change with more time: the policy rules are currently code rather than configuration, which means changing them requires a deploy. A declarative policy file with a schema would make the safe thing the easy thing for whoever operates it.

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      Provides AI assistants with the ability to control Linux desktop environments through tools for file management, application launching, and system operations like clipboard access. It includes a multi-level security model to manage permissions for safe, elevated, and restricted actions.
      6
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables full Linux desktop control including windows, mouse, keyboard, clipboard, audio, screenshots, OCR, accessibility, and system management through MCP-compatible AI agents.
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables an AI agent to see and control a Linux desktop via Wayland/Hyprland, providing screenshots, structured desktop state, pointer/keyboard input, semantic window/workspace tools, and a policy engine for safe execution.
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables tool-capable AI models to operate a private Linux desktop and headless browser, with terminal, file, and application controls, while keeping your physical mouse and keyboard separate.
      MIT