GhostDesk

Table of contents

• Why GhostDesk?

• How it works

• Quick start

• Secure local run (TLS + auth)

• Tools

• Model requirements

• From one agent to a workforce

• Configuration

• Security

• Troubleshooting

• Custom image

• License

Why GhostDesk?

Browser automation tools (Playwright, Puppeteer, Selenium…) were built for human test engineers driving a browser with selectors. They do one thing, and they do it well — inside the browser.

GhostDesk is built from the other end: for AI agents, driving everything a desktop runs. Browsers, native apps, IDEs, terminals, office suites, legacy software, internal tools. If it renders pixels on screen, your agent can see it and use it — in one conversation, across many applications, without a line of glue code.

You don't write selectors. You write a prompt:

"Open the CRM, export last month's leads as CSV, open LibreOffice Calc, build a pivot table, screenshot the chart, and email it to the team."

The agent opens the browser, logs in, downloads the file, switches to LibreOffice, processes the data, captures the result, composes the email, sends it. One prompt, multiple apps, fully autonomous — no glue code, no per-site scraper, no brittle selector chain.

That is what agents using a desktop looks like.

Runs on models you can actually host

Desktop control needs to be fast — an agent that takes twelve seconds to decide where to click is unusable. GhostDesk is tuned so that vision-language models from the Qwen family running on a single workstation GPU are a first-class target, not an afterthought. No API bill, no screenshots of your desktop leaving your network.

Frontier models (Claude, GPT-4o, Gemini) work too and remain the smoothest path — but they are not the bar. See Model requirements for the supported stacks and the one coordinate-space setting that matters.

How it works

GhostDesk runs a virtual Linux desktop inside Docker and exposes it as an MCP server. Your agent gets a sandboxed desktop with a taskbar, clock, and pre-installed applications — equivalent to what a human sees on their screen.

The agent perceives the screen by calling screen_shot(), which captures the full desktop at native resolution and returns it as WebP (or PNG). An optional region= argument can crop to a sub-rectangle when the agent explicitly wants to narrow its focus.

This works with any application — web apps, native apps, legacy software, Canvas, WebGL.

Quick start

1. Run the container

One command, plain HTTP, no password. Fine for kicking the tires on a laptop you trust — not fit for anything beyond that. Ready to harden it? Jump to Secure local run.

docker run -d --name ghostdesk-demo \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  ghcr.io/yv17labs/ghostdesk:latest

The latest image ships with Firefox, the foot terminal, mousepad (text editor), galculator, and passwordless sudo for the agent user — enough to demo a browsing + note-taking workflow out of the box. Need a different app set? Build your own on top of base — see Custom image.

The container boots in the dev posture: plain HTTP on both ports, every auth gate disarmed on purpose. You'll see warnings in the logs reminding you of that — they go away once you follow the secured path below.

2. Connect your AI

GhostDesk speaks MCP over the Streamable HTTP transport — any MCP-compatible client can drive it. Point your client at http://localhost:3000/mcp:

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}

Any other MCP-compatible client — same URL, no headers, no auth. That's the whole demo posture.

Drop in the recommended system prompt. SYSTEM_PROMPT.md gives your agent a battle-tested baseline — keyboard first, see/act/confirm, clear popups, scroll-to-read. It measurably improves reliability across both frontier and self-hosted models. Copy it into your agent's system prompt before the first run.

3. Watch your agent work

Open http://localhost:6080/ in your browser to see the virtual desktop in real time. No password prompt — the dev posture skips it.

Give your agent a first prompt to confirm the wiring is right:

"Take a screenshot of the desktop, list the installed applications, then open Firefox and go to wikipedia.org."

You should see Firefox launch in the noVNC tab, the URL bar fill in, and the page load — all under your agent's control.

4. When you're done

docker stop ghostdesk-demo && docker rm ghostdesk-demo

The demo run creates no named volume, so this leaves nothing behind.

Secure local run (TLS + auth)

The Quick start above drops every gate so you can kick the tires in thirty seconds. The moment you want to expose this to anything beyond your own laptop — another machine on your LAN, a devcontainer port-forward on an untrusted network, a teammate's browser — flip to the secured posture: real TLS + bearer-token auth on MCP + password prompt on noVNC.

GhostDesk couples TLS and auth: mount a cert and you get wss:// + bearer-token on MCP + a single-password prompt on noVNC (see Security → Auth ≡ TLS). mkcert issues a browser-trusted cert for localhost in two commands:

# Issue a locally-trusted cert (first time only — installs a local CA in your trust store)
mkcert -install
mkdir -p tls
mkcert -cert-file tls/server.crt -key-file tls/server.key localhost 127.0.0.1 ::1

# Generate the MCP and VNC secrets
export GHOSTDESK_AUTH_TOKEN=$(openssl rand -hex 32)
export GHOSTDESK_VNC_PASSWORD=$(openssl rand -hex 16)

Pick a container name that matches the agent's role — sales-agent, research-agent, accounting-agent… Below we use my-agent as a placeholder; replace it everywhere in the command.

# Run the container — cert mounted, TLS + auth enabled everywhere
docker run -d --name ghostdesk-my-agent \
  --restart unless-stopped \
  --cap-add SYS_ADMIN \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  -v ghostdesk-my-agent-home:/home/agent \
  -v "$PWD/tls/server.crt:/etc/ghostdesk/tls/server.crt:ro" \
  -v "$PWD/tls/server.key:/etc/ghostdesk/tls/server.key:ro" \
  -e GHOSTDESK_AUTH_TOKEN \
  -e GHOSTDESK_VNC_PASSWORD \
  -e TZ=America/New_York \
  -e LANG=en_US.UTF-8 \
  ghcr.io/yv17labs/ghostdesk:latest

echo "MCP token:    $GHOSTDESK_AUTH_TOKEN"
echo "VNC password: $GHOSTDESK_VNC_PASSWORD"

Once the container is up, update your MCP client config — same shape as the demo, now over https:// with a bearer token:

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "https://localhost:3000/mcp",
      "headers": {
        "Authorization": "Bearer <paste $GHOSTDESK_AUTH_TOKEN here>"
      }
    }
  }
}

Any other MCP-compatible client — same URL, plus an Authorization: Bearer <token> header in whatever form your client accepts.

Then open https://localhost:6080/ in your browser — the mkcert CA installed by mkcert -install is already in your trust store, so the browser accepts the cert with no warning. noVNC will prompt for $GHOSTDESK_VNC_PASSWORD.

Going to production? Swap the mkcert leaf for a real cert, source both secrets from your secret manager, and front port 6080 with an identity-aware proxy — SECURITY.md has the full contract.

--cap-add SYS_ADMIN — Required by Electron apps (VS Code, Slack, etc.) and other applications that need Linux user namespaces to run their sandbox. Safe to remove if you don't need them.

The named volume persists the agent's home directory across restarts — browser passwords, bookmarks, cookies, downloads, and desktop preferences are all preserved. On the first run, Docker automatically seeds the volume with the default configuration from the image.

Tools

12 tools at your agent's fingertips, grouped by concern (verb_noun naming):

Screen

Mouse

Keyboard

Clipboard

Apps

Model requirements

Your inference stack must cover four capabilities — all four are mandatory:

1. Text + vision — the agent perceives the desktop through screenshots and needs a model that can interpret them.

2. Tool use — GhostDesk exposes 12 tools as function calls; the model must be able to invoke them.

3. MCP client — the host needs to speak Streamable HTTP MCP to reach the GhostDesk server.

4. WebP image support — GhostDesk returns screenshots as WebP by default to keep payloads small and inference fast. A stack that can only decode PNG or JPEG will not work out of the box.

Coordinate space — GhostDesk-Model-Space header

By default no header is needed: Claude and the other major frontier LLMs work out of the box. Qwen3.5 and Qwen3-VL need the client to send GhostDesk-Model-Space: 1000 on every MCP request.

Example MCP client config:

{
  "mcpServers": {
    "ghostdesk": {
      "url": "https://localhost:3000/mcp",
      "headers": {
        "GhostDesk-Model-Space": "1000"
      }
    }
  }
}

Running locally

For self-hosted inference we use and recommend our fork of llama.cpp, which adds WebP decoding on top of upstream: YV17labs/llama.cpp. The day WebP lands upstream we will archive the fork and point there directly.

macOS users: use llama.cpp, not mlx-vlm (as of 2026-04-01). The mlx-vlm stack currently produces inaccurate coordinate outputs for the same models that work correctly under llama.cpp. This is caused by an upstream bug in an Apple dependency, not the model itself. Until the fix lands, llama.cpp is the recommended backend on every platform — including Apple Silicon Macs.

Run whatever local model you like. Three from the Qwen vision family that I've used and that work well for desktop control:

• Qwen3.6-35B-A3B — 35B parameters, only 3B active per token.

• Qwen3.5-35B-A3B — 35B parameters, only 3B active per token.

• Qwen3-VL — the Qwen3 vision-language branch, available in several sizes on the Qwen Hugging Face org.

From one agent to a workforce

Each GhostDesk instance is a container. Spin up one, ten, or a hundred — each agent gets its own isolated desktop, its own apps, its own role. Think of it as hiring a team of digital employees, each with their own workstation.

Scale horizontally

# docker-compose.yml — 3 specialized agents, one command
#
# Prerequisites: the TLS cert + key at ./tls and the two secrets
# (GHOSTDESK_AUTH_TOKEN, GHOSTDESK_VNC_PASSWORD) in your environment or a
# .env file. Generate both exactly as shown in the Secure local run
# section above. See SECURITY.md for the production secret-handling
# contract.

x-ghostdesk-defaults: &ghostdesk-defaults
  image: ghcr.io/yv17labs/ghostdesk:latest
  restart: unless-stopped
  cap_add: [SYS_ADMIN]
  shm_size: 2g
  environment:
    - GHOSTDESK_AUTH_TOKEN
    - GHOSTDESK_VNC_PASSWORD
    - TZ=America/New_York
    - LANG=en_US.UTF-8

services:
  sales-agent:
    <<: *ghostdesk-defaults
    container_name: ghostdesk-sales-agent
    ports: ["3001:3000", "6081:6080"]
    volumes:
      - ghostdesk-sales-agent-home:/home/agent
      - ./tls/server.crt:/etc/ghostdesk/tls/server.crt:ro
      - ./tls/server.key:/etc/ghostdesk/tls/server.key:ro

  research-agent:
    <<: *ghostdesk-defaults
    container_name: ghostdesk-research-agent
    ports: ["3002:3000", "6082:6080"]
    volumes:
      - ghostdesk-research-agent-home:/home/agent
      - ./tls/server.crt:/etc/ghostdesk/tls/server.crt:ro
      - ./tls/server.key:/etc/ghostdesk/tls/server.key:ro

  accounting-agent:
    <<: *ghostdesk-defaults
    container_name: ghostdesk-accounting-agent
    ports: ["3003:3000", "6083:6080"]
    volumes:
      - ghostdesk-accounting-agent-home:/home/agent
      - ./tls/server.crt:/etc/ghostdesk/tls/server.crt:ro
      - ./tls/server.key:/etc/ghostdesk/tls/server.key:ro

volumes:
  ghostdesk-sales-agent-home:
  ghostdesk-research-agent-home:
  ghostdesk-accounting-agent-home:

docker compose up -d   # Your workforce is ready

Each agent runs in parallel, independently, on its own desktop. Connect each to a different LLM, give each a different system prompt, install different apps — full specialization.

Secure by design

Every agent is sandboxed in its own container. No access to the host machine. No access to other agents. Network, filesystem, and process isolation come free from Docker.

This makes GhostDesk a natural fit for enterprises:

Specialize each agent

Give each agent a role, like you would a new hire:

• Sales agent — monitors the CRM, enriches leads, updates the pipeline

• Research agent — browses the web, compiles competitive intelligence, writes reports

• Accounting agent — processes invoices in legacy ERP software, reconciles spreadsheets

• QA agent — clicks through your app, files bug reports with screenshots

• Support agent — handles tickets, looks up customer info across multiple internal tools

Each agent gets its own system prompt defining its mission, its own installed applications, and its own network permissions. Manage AI agents like employees — each with their own desktop, their own tools, and their own clearance level.

Supervise in real time

Every agent exposes a VNC/noVNC endpoint. Open a browser tab and watch your agent work — or open ten tabs and monitor your entire workforce. Intervene at any time: take over the mouse, correct course, or chat with the orchestrating LLM.

Configuration

Every variable GhostDesk reads is namespaced under GHOSTDESK_*. Standard POSIX variables (TZ, LANG) are kept as-is so the existing Unix ecosystem keeps working.

Secrets (required — container refuses to boot without them)

Both are plain environment variables. Wire them from your secret store (secretKeyRef on Kubernetes, Docker secrets / Vault / AWS SM on compose) — see SECURITY.md for the full contract.

Runtime knobs

Pinned values (not configurable)

Security

GhostDesk owns two things: transport encryption and authentication. Everything else (rate limiting, SSO, WAF, session recording, brute-force protection, per-user identity on noVNC) is a reverse-proxy concern — the container is designed to run behind one, not directly on the internet.

The full threat model, the Auth ≡ TLS posture switch, the wayvnc RFB-type-2-inside-wss:// rationale, the secrets handling contract, and the exhaustive in-scope / out-of-scope table all live in SECURITY.md — single source of truth. Start there before deploying to anything you don't fully trust.

Reporting a vulnerability? Use GitHub's private security advisory — see SECURITY.md § Reporting.

Troubleshooting

My agent's clicks land off-target by a huge margin

Almost always a coordinate-space mismatch. Frontier models (Claude, GPT-4o, Gemini) need no header (default pass-through); the Qwen vision family needs the client to send GhostDesk-Model-Space: 1000 on every MCP request. Full rationale in Model requirements → Coordinate space.

The container refuses to start with a secrets error

The prod posture (cert mounted) requires both GHOSTDESK_AUTH_TOKEN and GHOSTDESK_VNC_PASSWORD to be set — GhostDesk refuses to boot without them on purpose, to prevent an unauthenticated prod container. Generate them as shown in Secure local run and pass them with -e. The demo posture (no cert) has no such requirement.

noVNC shows a black screen or the desktop renders with graphical glitches

You're probably short on shared memory. Browsers and other GPU-accelerated apps inside the container need a reasonable /dev/shm — --shm-size 2g is the baseline in every example and should not be trimmed. If you already have --shm-size 2g, check the container logs for wayvnc or compositor errors.

Firefox / Electron apps fail to launch or crash immediately

Electron-based apps (VS Code, Slack, Discord…) need Linux user namespaces for their sandbox. Add --cap-add SYS_ADMIN to your docker run (already present in the Secure local run example). Firefox itself works without it.

Custom image

The base tag provides GhostDesk without any pre-installed GUI application — just the virtual desktop, VNC, and the MCP server. Use it to build your own image with only the tools you need:

FROM ghcr.io/yv17labs/ghostdesk:base

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        chromium-browser \
        libreoffice-calc \
    && rm -rf /var/lib/apt/lists/*

docker build -t my-agent .

See the project's Dockerfile for a complete example.

License

Functional Source License, Version 1.1, ALv2 Future License (FSL-1.1-ALv2) — see LICENSE for the authoritative terms.

What this means in practice (informal summary — the LICENSE file governs; this is not legal advice):

• Permitted purposes cover the use cases that matter for the vast majority of users: internal use and access inside your company, non-commercial education and research, and professional services you provide to a licensee who is using GhostDesk in accordance with the license. Self-hosting GhostDesk to run your own agents — even commercial, revenue-generating workflows that power your product — is a permitted internal use.

• Competing Use is prohibited. You may not make GhostDesk available to others in a commercial product or service that substitutes for GhostDesk, substitutes for any product or service the project offers using GhostDesk, or provides the same or substantially similar functionality. In short: you cannot take GhostDesk and rebrand it, host it as a paid service, or build a competing desktop-automation-for-agents product from it.

• Apache 2.0 in two years. Each released version of GhostDesk becomes available under the Apache License 2.0 on the second anniversary of its release, automatically and irrevocably. The Competing Use restriction only applies for those first two years.

Commercial licensing. If your intended use falls under Competing Use — you want to resell GhostDesk, offer it as a managed service, or build a competing product — contact the maintainers to discuss a commercial license before deploying. Open a GitHub issue or reach out directly; we are happy to talk.