Skip to main content
Glama

own-computer-use

License Platform Python Tools PyPI CI

Demo

A self-hosted MCP server that gives AI agents real control of a Windows PC: screen reading with built-in OCR (Arabic and English), mouse, keyboard, clipboard, and app launching. It runs on any plan, with any client, and blocks every action that could affect the outside world unless you explicitly allow it.

Works with Claude Code, Claude Desktop, Codex, Cursor, or any MCP client. Python only, no cloud component, no telemetry.

Why this exists

Desktop computer-use features in commercial AI clients are locked behind plans and platforms. The alternative is to host the capability yourself. This is a pair of Python files speaking MCP over stdio. Point your client at it and your machine becomes the tool surface. If you pay per token through your own gateway (ANTHROPIC_BASE_URL), it works the same.

Related MCP server: windows-computer-use-mcp

What the agent can actually do

This is full-machine control, not a browser extension. Anything a person does with a mouse and keyboard, the agent does through these 21 tools.

Every installed program. Photoshop, CapCut, Premiere, Excel, Word, PowerPoint, OBS, game launchers, your own builds. computer_open_application starts them and reports what actually happened; clicks, typing, and hotkeys drive them.

Video and photo editing. Cut clips in CapCut or Premiere, drag timeline handles with press-hold (computer_mouse_down_up), apply filters through menus that need millisecond timing (computer_batch), export the result.

Spreadsheets and documents. Fill Excel cell by cell, format in Word, assemble slide decks, trigger macros with computer_key.

Messaging and email. Read a thread with OCR, draft a reply, send it once you approve. Reading needs no consent; sending always does.

Web apps. Facebook, Telegram Web, dashboards, admin panels, anything behind a login. OCR reads Arabic and English text straight off the screen, so the agent acts on pages that DOM tooling cannot reach.

System work. File Explorer, Control Panel dialogs, installers, settings panes. If it renders on your screen, it is reachable.

Anything with a GUI, and anything without one. The agent uses a terminal or API for file and data work when that is faster, and switches to the GUI for the parts that need a real window. computer_selfcheck keeps the whole surface healthy between runs.

Writing quality is a separate skill. When the agent drafts personal messages on your behalf, pair it with the humanizer skill in Claude Code: replies in Messenger, WhatsApp, or email come out in a natural human voice instead of sounding like a model, and you read them before anything is sent.

The 21 tools

Category

Tools

See

computer_screenshot (entire virtual screen, auto-scaled to model limits) · computer_zoom (magnify a region; coordinates stay in full-screenshot space) · computer_read_screen (built-in OCR, Arabic and English, optional atomic target_title focus+read)

Act (consent-gated)

computer_click (left/right/middle, single/double/triple) · computer_type (clipboard paste, Arabic and Unicode safe) · computer_key · computer_hold_key · computer_scroll · computer_drag · computer_mouse_down_up (press-hold for sliders and multi-select)

Batch (consent-gated)

computer_batch — several actions milliseconds apart in one call, for timing-critical UI: menus that close on blur, mid-flight buttons

Multi-monitor

computer_capture_monitor (screenshot ONE physical display 1..N) · computer_display_overview (list every monitor: index, resolution, virtual-desktop position)

Teach

computer_teach_step (scaffold: human performs an action, agent observes and replays it as a batch sequence)

System

computer_open_application (consent-gated, injection-guarded) — honest launch report: opened, crashed on startup, no window after 8s · computer_focus_window (bring any window forward by title substring) · computer_clipboard (read/write) · computer_cursor_position · computer_wait

Agents

computer_selfcheck (verify + auto-repair imports, audit log, capture, OCR, gate) · computer_review (audit the last 30 actions against your stated task, return CLEAN or REVIEW NEEDED)

Built on FastMCP, tuned for speed

The server is a FastMCP stdio server, so it plugs into every MCP client that speaks the standard protocol, and FastMCP handles tool schemas, validation, and error shaping for you.

Speed was a design constraint, not an afterthought. On a 2011 dual-core laptop (the machine this was built on), a naive version of the same server took over a minute to answer its first call.

Optimization

Effect

Lazy imports

tools/list answers without loading pyautogui, mss, or the OCR engine

Boot warm-up thread

Heavy libraries load in the background while the MCP handshake runs, so the first real call lands in ~8s instead of 60s+

Full virtual screen capture

One mss grab of the whole desktop; no per-monitor round trips

Screenshot downscale in-process

PIL resize at capture time, never on the model side

One-shot tool calls

computer_batch removes the model round trip between actions (seconds to milliseconds)

Cached OCR engine

RapidOCR is instantiated once per server, not per read

The result: a first call in seconds, subsequent calls in milliseconds, and a batch of ten actions in one model turn.

Safety, enforced in code

Three gates sit between an agent and your machine.

Self-harm guard. Typing and key presses are blocked while an agent surface holds focus (Claude, VS Code, terminals). Text aimed at a chat window would land in the conversation itself, or execute as a shell command.

Consent gate. Every machine-mutating tool is blocked at server level unless a file named CONSENT.txt exists next to the server, starts with ALLOW, and is less than 10 minutes old:

ALLOW send-reply-to-messenger

Blocked without it: click, type, key (Enter), scroll, drag, mouse down/up, hold key, batch, open_application. Free: screenshot, zoom, OCR, cursor position, clipboard read.

The gate auto-expires. A misaligned model cannot click Submit, type into your apps, or send messages unless you deliberately created that file within the last ten minutes. computer_open_application also rejects shell metacharacters (& | ^ < > ! % ") before any process starts, and OCR output is returned verbatim, never executed.

Every claim in this section is verified by a script you can run: python test_security.py (13-tool matrix) and python test_security.py --injection (6 payloads). The full model, including what the gates do NOT protect against, is in SECURITY.md.

Audit log. Every executed action lands in actions.log with a timestamp and its arguments (1 MB rotation). computer_review turns the log into a verdict. A failsafe on the host side: jamming the mouse into a screen corner aborts input injection.

Installation

Windows, Python 3.10+.

One line (recommended) — from PyPI, no clone needed:

claude mcp add own-computer-use --scope user -- uvx own-computer-use-mcp
claude mcp list    # expect: own-computer-use ... Connected

Add OCR support to the installed package:

uvx --with rapidocr-onnxruntime own-computer-use-mcp

From source — if you want to edit the code:

git clone https://github.com/3mad101c/own-computer-use.git
cd own-computer-use
pip install -r requirements.txt
claude mcp add own-computer-use --scope user -- python C:\path\to\mcp_server.py
claude mcp list    # expect: own-computer-use ... Connected

Verify the safety gates yourself (see SECURITY.md):

python test_security.py               # 13/13: 9 mutating tools blocked, 4 read-only free
python test_security.py --injection   # 6/6: shell metacharacters rejected

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "own-computer-use": {
      "command": "python",
      "args": ["C:\\path\\to\\mcp_server.py"]
    }
  }
}

OCR is optional. Without rapidocr_onnxruntime everything else works; install it for computer_read_screen:

pip install rapidocr-onnxruntime

Worked examples

Edit a video. "Open my CapCut project in Downloads, cut the clip at 00:12, export it."

The agent launches CapCut, reads the timeline with OCR, clicks the split point, exports, and reports what happened at each step. Clicking the export button needs a live CONSENT.txt; everything up to it does not.

Fill a spreadsheet. "Open budget.xlsx, put last month's totals in column D, bold the header."

Excel launches, the agent types into cells, uses computer_key for formatting, and saves with Ctrl+S.

Handle a message. "Read my last message on Messenger and reply that I'll call later."

computer_read_screen reads the thread, the agent drafts in a human voice (with the humanizer skill), shows you the text, and only sends once you approve and CONSENT.txt is live.

Review what happened. "Review the last 30 actions against the task I gave you."

computer_review returns CLEAN or REVIEW NEEDED, listing every action with real-world effects.

Standalone CLI agent

agent.py runs the same computer-use loop outside MCP: it streams Anthropic's computer_toolset_20260801 on claude-opus-5-5 and executes each requested action locally, honoring the same consent file.

python agent.py "open calculator and compute 7*8"

It needs ANTHROPIC_API_KEY in the environment. It also honors ANTHROPIC_BASE_URL, so your own gateway works.

Architecture

A few decisions worth knowing before you extend it.

Coordinate space. Screenshots are downscaled to a 1568 px long edge. The scale factor is tracked, and every coordinate the model sends in screenshot space is mapped back to physical pixels inside screen.py. The model never sees your resolution or DPI.

Full virtual screen. Capture uses mss monitor 0 (the entire virtual desktop), so an app opening on a second display cannot produce a screenshot of the wrong screen.

Files:

mcp_server.py   FastMCP stdio server: 21 tools, guards, consent gate, OCR
screen.py       capture, coordinate mapping, input actions, audit log
agent.py        optional standalone Claude-API computer-use loop
skills/         SKILL.md teaching agents the look-act-look discipline
test_all_tools.py  stdio integration test for the whole tool surface

Known limitations

Windows only (Win32 focus and input injection). macOS and Linux ports are welcome.

Elevated applications (Task Manager, UAC prompts) reject synthetic input by Windows UIPI design.

RapidOCR reads English well and Arabic acceptably; treat it as a reading aid and confirm critical text with a zoomed screenshot.

IME layouts can alter computer_key and computer_hold_key (virtual keycodes pass through the IME). Clipboard-paste typing is immune.


بالعربي

سيرفر MCP مجاني ومفتوح المصدر يدي أي وكيل ذكاء اصطناعي تحكم كامل في جهاز ويندوز بتاعك: شاشة، ماوس، كيبورد، وقراءة نص من الصورة بالعربي والإنجليزي. بيشتغل مع Claude Code وClaude Desktop وCodex وCursor وأي عميل MCP — على أي خطة، وبوابة خاصة لو عندك.

بيعمل إيه؟ أي حاجة بتعملها بماوس وكيبورد: مونتاج في CapCut أو Premiere، جداول في Excel، ردود على رسايل، تشغيل أي برنامج، وحتى إعدادات النظام. 21 أداة MCP تغطي الشاشة والتحكم والقرارات.

أمان أول: كل أداة بتغيّر حاجة على الجهاز (نقر، كتابة، سكرول، فتح برنامج) مقفولة إلا لو أنت أنشأت ملف CONSENT.txt فيه كلمة ALLOW — وصلاحيته 10 دقايق بس. القراءة واللقطات شغالة بحرية.

git clone https://github.com/3mad101c/own-computer-use.git
cd own-computer-use && pip install -r requirements.txt
claude mcp add own-computer-use --scope user -- python C:\path	o\mcp_server.py

الترخيص MIT. المشروع بُني واختُبر على ويندوز 11.

License

MIT

Available Tools

21 tools
computer_batchComputer BatchA

Run several actions milliseconds apart in ONE call — for timing-critical UI (menus that close on blur, mid-flight buttons). Each item: {"action": , "input": }. Coordinates follow the same screenshot-space rule. Example: [{"action":"left_click","input":{"x":100,"y":50}}, {"action":"key","input":{"key":"Down"}},{"action":"key","input":{"key":"Return"}}]. Returns one result line per action.

Inputs: actions (required). Returns: result

computer_capture_monitorComputer Capture MonitorA

Screenshot ONE physical monitor (1..N). computer_screenshot captures the whole virtual desktop; use this to zoom on one display on multi-monitor setups. List displays with computer_display_overview.

Inputs: monitor.

computer_clickComputer ClickA

Click at (x, y) in the LAST screenshot's pixel space. button: left|right|middle, clicks: 1=single, 2=double, 3=triple (select line/paragraph). Outbound gate: a click can press Submit/Send/Delete — needs fresh CONSENT.txt.

Inputs: x (required), y (required), button, clicks. Returns: result

computer_clipboardComputer ClipboardA

Read or write the system clipboard: action read|write. write copies text; read returns current clipboard content (empty string if none).

Inputs: action (required), text. Returns: result

computer_cursor_positionComputer Cursor PositionB

Current mouse cursor position in SCREENSHOT coordinates.

Inputs: no named parameters. Returns: result

computer_display_overviewComputer Display OverviewA

List all physical monitors with resolution and virtual-desktop position. Returns JSON array: index, left, top, width, height. Use before computer_capture_monitor to pick the right display index.

Inputs: no named parameters. Returns: result

computer_dragComputer DragB

Drag from (start_x, start_y) to (x, y) — file moves, sliders, selection.

Inputs: x (required), y (required), start_x (required), start_y (required). Returns: result

computer_focus_windowComputer Focus WindowA

Bring the first visible top-level window whose title contains title_substring (case-insensitive) to the foreground. Returns the exact title focused or 'not found'. Keyboard actions land there afterward.

Inputs: title_substring (required). Returns: result

computer_hold_keyComputer Hold KeyA

Hold a key for duration seconds then release (e.g. Shift for click-select ranges). One chord only — no '+' combos.

Inputs: key (required), duration. Returns: result

computer_keyComputer KeyB

Press a key or chord: "Return", "ctrl+c", "alt+Tab", "escape". Emphasis: Enter/Return in a chat app SENDS — that needs consent too.

Inputs: key (required). Returns: result

computer_mouse_down_upComputer Mouse Down UpA

Hold or release the LEFT mouse button without moving: action down|up. For press-drag-press-drag-... sequences pyautogui can't do in one call.

Inputs: action. Returns: result

computer_open_applicationComputer Open ApplicationA

Open an app by name or full .exe path and report what REALLY happened: opened (window appeared) / running-no-window / crashed / nothing-launched. Windows: use start semantics, e.g. "notepad" or "chrome". Waits up to 8s for a visible window.

Inputs: target (required). Returns: result

computer_read_screenComputer Read ScreenA

OCR the screen. Optional target_title: bring the first window whose title contains it to the foreground FIRST (atomic focus+read — the read cannot miss the target). Returns lines "(x,y) text". Arabic + English.

Inputs: target_title. Returns: result

computer_reviewComputer ReviewA

Reviewer agent: audits the LAST actions in actions.log against the stated task_description. Flags: actions with external effects that ran, consent-gate bypasses, actions not plausibly related to the task, and error-results. Returns findings + verdict.

Inputs: task_description. Returns: result

computer_screenshotComputer ScreenshotA

Take a screenshot of the entire virtual screen (all monitors). Returns the image; its pixel space defines the coordinate system for all computer_* tools.

Inputs: no named parameters.

computer_scrollComputer ScrollB

Scroll at (x, y). direction: up|down|left|right; amount in wheel notches.

Inputs: x (required), y (required), amount, direction. Returns: result

computer_selfcheckComputer SelfcheckA

Self-heal agent: verify all server subsystems and FIX what it can. Checks heavy imports, audit log writability, capture (non-blank), OCR engine load, screenshots scale sanity, outbound gate state. Returns per-subsystem status + list of automatic repairs applied.

Inputs: no named parameters. Returns: result

computer_teach_stepComputer Teach StepA

Teach mode (stub scaffold): records that the HUMAN wants to demo an action for the agent to repeat. Returns guidance text. Full playback recording is planned; use computer_batch scripts for repeatable sequences today.

Inputs: instruction (required). Returns: result

computer_typeComputer TypeA

Type text into the focused control via clipboard paste (Arabic-safe). Outbound gate: requires fresh CONSENT.txt (ALLOW line, <10 min old) — typing sends text into an app the user may not see being typed into.

Inputs: text (required). Returns: result

computer_waitComputer WaitA

Pause up to 10s between actions (page loads, animations).

Inputs: seconds. Returns: result

computer_zoomComputer ZoomA

Zoom: re-capture a region at higher magnification for reading small text. Coordinates stay in the FULL-screenshot pixel space, never the crop's.

Inputs: x (required), y (required), width (required), height (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 21 tools

    Disambiguation5/5

    Each tool has a clearly distinct role: input actions (click, type, key, scroll, drag), screen inspection (screenshot, zoom, read_screen), display management, and meta/system tools are well separated. The several capture-related tools are distinguished by scope (whole virtual screen vs one monitor vs region zoom vs display listing) and their descriptions make the intended usage explicit.

    Naming Consistency5/5

    All tools use a consistent `computer_` prefix with snake_case names, and there is no mixing of camelCase or unrelated conventions. Some names are noun-like (`computer_screenshot`, `computer_clipboard`) rather than pure verb_noun, but the pattern is predictable throughout.

    Tool Count4/5

    At 21 tools, the server is slightly above the typical well-scoped range, but most tools earn their place by covering distinct GUI automation concerns. A few meta/extras such as `computer_teach_step` feel less essential, but the overall count is reasonable for full computer-use coverage.

    Completeness4/5

    The surface covers the core GUI lifecycle well: screen capture, mouse/keyboard input, clipboard, application launch, window focus, OCR, batch actions, and safety/review checks. Minor gaps remain, such as no explicit window listing, minimize/close/resize operations, or standalone right-button hold/release, but agents can work around these in most cases.

    Maintenance

    ActivityMaintained
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that gives AI agents human-like control over Windows via visual perception and simulated mouse and keyboard input, enabling automation of any application without APIs.
      33 npm
      6
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      A local, dependency-free MCP server that gives AI agents controlled access to the active Windows desktop, enabling automated interaction with applications through screenshots, clicks, typing, and window management.
      1
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      A local MCP server that lets multiple AI agents safely share a Windows desktop by coordinating exclusive access to the mouse, keyboard, screen, and windows, with human consent, takeover, and full activity logging.
      10 npm
      MIT