taw-computer

What if your AI could do everything you do on a computer?

Not just write code — but open a browser, click buttons, fill forms, run servers, test in real browsers, install anything, and see the screen?

taw-computer is an open-source MCP server that gives AI agents a full Ubuntu desktop inside Docker. Your AI connects, gets a real computer, and works like a human would.

No internal LLM. No chat UI. Your AI is the brain. This is the body.

Demo

Why taw-computer?

Other tools let AI write code. taw-computer lets AI use a computer.

Quick start

Get running in under 5 minutes:

# 1. Clone & build
git clone https://github.com/the-agents-work/taw-computer.git
cd taw-computer
docker build -f images/Dockerfile.taw -t taw-computer-base .

# 2. Install & start
npm install && npm start

Then add to your AI client:

{
  "mcpServers": {
    "taw-computer": {
      "command": "npx",
      "args": ["tsx", "/path/to/taw-computer/mcp/index.ts"]
    }
  }
}

Add to Cursor MCP settings (Settings → MCP Servers) — same JSON format as above.

Add to claude_desktop_config.json — same JSON format as above.

taw-computer speaks standard MCP over stdio. Any client that supports MCP can connect.

Got a powerful server / Mac Mini / VPS? Run taw-computer there and connect from anywhere:

{
  "mcpServers": {
    "taw-computer": {
      "command": "ssh",
      "args": ["user@your-server", "cd /path/to/taw-computer && npx tsx mcp/index.ts"]
    }
  }
}

Your laptop (Claude Code)
    ↕ SSH (stdin/stdout piped over network)
Remote server (taw-computer + Docker)
    ↕ Docker
Ubuntu sandbox

Setup:

1. On the server: install Docker, clone repo, build image, npm install

2. On the server: enable SSH (sudo systemctl enable ssh)

3. On your laptop: ssh-copy-id user@your-server (passwordless login)

4. Add the MCP config above — done!

Watch via VNC: open http://your-server:6080 in your browser.

That's it. Now tell your AI: "Create a VM and build me a website" — and watch it work.

What can it do?

🖥️ "Build me a landing page"

AI creates a VM → scaffolds Next.js → writes components → starts dev server → opens browser to check → iterates until it looks right

🌐 "Go to Amazon and find the best laptop under $1000"

AI opens Chromium → navigates to Amazon → searches → scrolls → extracts prices → compares → reports back

🧪 "Run E2E tests on my deployed app"

AI launches Playwright → navigates to your URL → fills forms → clicks buttons → asserts results → reports failures

🔧 "Set up a PostgreSQL database with sample data"

AI runs apt install postgresql → creates database → writes seed script → runs it → verifies with queries

📸 "What does my app look like on mobile?"

AI takes desktop screenshot → resizes viewport → screenshots again → compares → suggests CSS fixes

How it works

┌─────────────────────────────────────────────────────┐
│  Your AI Client                                     │
│  Claude Code · Cursor · Claude Desktop · any MCP    │
└───────────────────────┬─────────────────────────────┘
                        │ MCP protocol (stdio)
┌───────────────────────▼─────────────────────────────┐
│  taw-computer MCP server          30+ tools         │
│  vm · shell · files · browser · desktop · search    │
└───────────────────────┬─────────────────────────────┘
                        │ Docker API
┌───────────────────────▼─────────────────────────────┐
│  Ubuntu 22.04 Sandbox              isolated container│
│                                                      │
│   bash    Chromium + CDP    xfce4 Desktop + VNC     │
│     git npm pip curl          Playwright              │
│       python node              xdotool scrot          │
│                                                      │
│   /workspace ← your project files live here          │
└──────────────────────────────────────────────────────┘

30+ tools

Set-of-Mark: how browser automation actually works

Most "computer use" tools guess pixel coordinates. We use Set-of-Mark prompting — the AI sees numbered badges on every interactive element:

Step 1: browser_snapshot
        → AI sees screenshot with [1] Login  [2] Search  [3] Cart  ...

Step 2: browser_click_ref(ref=2)
        → clicks the Search box precisely

Step 3: browser_type_ref(ref=2, text="laptop", submit=true)  
        → types and presses Enter

Step 4: browser_snapshot
        → sees new page with results [4] [5] [6] ...

No coordinate guessing. No CSS selector fragility. The AI sees what it's clicking.

VNC — watch your AI work in real time

Every sandbox comes with a noVNC web viewer. Open the URL in your browser and watch:

• 🖱️ AI navigating websites and clicking buttons

• ⌨️ AI writing code in the terminal

• 🏗️ AI building and testing applications

• 🐛 AI debugging by inspecting the screen

Perfect for demos, debugging, and building trust in AI agents.

What's inside each sandbox

Configuration

Requirements

Project structure

taw-computer/
├── mcp/
│   ├── index.ts            # MCP server — stdio, 30+ tool handlers
│   └── browser.ts          # Playwright CDP + Set-of-Mark engine
├── sandbox/
│   ├── SandboxManager.ts   # Abstract interface
│   ├── DockerSandbox.ts    # Docker implementation
│   ├── FirecrackerSandbox.ts # Firecracker microVM (optional)
│   ├── NetworkManager.ts   # Network isolation
│   ├── config.ts           # Env-based config
│   └── index.ts            # Auto-detect backend
├── images/
│   └── Dockerfile.taw      # Ubuntu sandbox image
├── .github/
│   ├── workflows/ci.yml    # CI: typecheck + Docker build
│   └── ISSUE_TEMPLATE/     # Bug report + feature request
├── package.json
├── CONTRIBUTING.md
└── LICENSE (MIT)

Contributing

We'd love your help! See CONTRIBUTING.md.

Ideas for first contributions:

• 🎨 Record a demo GIF for this README

• 📝 Write a tutorial ("Build X with taw-computer")

• 🔧 Add a new MCP tool (audio? clipboard? multi-tab?)

• 🐳 Build a slimmer Docker image

• 🧪 Add automated tests

• 📦 Support Podman / containerd

Hosted version

Don't want to self-host? shipkit.cc — managed taw-computer with:

• Chat UI (just type what you want)

• Auth & team collaboration

• One-click app sharing

• No Docker setup needed

FAQ

Those are closed-source, hosted-only products that generate code. taw-computer gives AI a real computer — it can run servers, browse the web, install anything, and interact with any desktop app. It's also open source and self-hostable.

OpenInterpreter runs code on your local machine (risky). Open Hands uses its own LLM orchestration. taw-computer is just the computer — no built-in LLM, no opinions about orchestration. Your existing AI client (Claude Code, Cursor, etc.) is the brain. taw-computer is a pure MCP server.

Each sandbox is an isolated Docker container with its own filesystem, network, and process space. Nothing inside can touch your host system. Containers have memory/CPU/PID limits. When you're done, destroy the VM.

Yes — any AI client that supports MCP can connect. The server doesn't care which LLM is behind the client.

Yes. Anywhere Docker runs, taw-computer runs. The sandbox image supports both arm64 (Apple Silicon) and amd64 (Intel/AMD).

Star History

Related

• Model Context Protocol — the protocol taw-computer speaks

• Claude Code — AI coding CLI that supports MCP

• Playwright — browser automation powering CDP tools

License

MIT — do whatever you want with it.