Clickwarden

Windows computer use for AI agents, accessibility first. Clickwarden reads the Windows UI Automation tree to find and click real controls, and asks a vision model of your choice only when a control isn't exposed, so the agent works from short text results instead of screenshots. It drives your real mouse and keyboard, with a visible control panel and hard stops, and exposes a text-only MCP server and a CLI. Open source, Apache-2.0.

Phone recording of an agent typing in a blank Google Doc, opening Drive and Gmail in new tabs by typing their addresses, and returning to the document. Recorded with an earlier internal build, when the project was still called Winflash (the name on the panel). The inbox and file lists are blurred on purpose.

⚠️ Read this before you run it

Clickwarden is invasive by design. While a session runs it moves your real cursor, clicks and types on your real desktop, as you, in whatever application is in front. A mistake, a wrong click or a malicious web page can send messages, delete files, spend money or change settings in accounts you are signed in to.

Your screen leaves your computer. Screenshots of the working window are sent to the vision provider you choose. Everything visible in those pixels (private messages, names, balances, one-time codes) goes to that service, which may store it under its own privacy policy. Pick a local model (Ollama, LM Studio) to keep it all on your machine.

You are responsible for what you automate. Use it only on computers and accounts you are allowed to operate. Keep it away from banking, password managers and other people's private data. Do not leave a session unattended. Details: SAFETY.md.

Status: beta (v0.1.0). Windows 10/11 only. No warranty (see LICENSE).

Related MCP server: openowl

About this project (the honest version)

I am Kaan, a second-year computer engineering student, and this is a hobby project. I built it by "vibe coding": directing AI coding assistants and testing the results, not by writing every line myself or having it professionally reviewed. AI agents that operate a computer ("computer use") are a very recent topic, and I wanted to learn how they work and contribute something open instead of just watching. I have run the offline test suite and tried it on my own machine, but that is all the assurance there is.

So please treat it as an experiment: read the code, expect rough edges, do not trust it with anything you cannot afford to lose, and tell me what is wrong. Issues, corrections and criticism are welcome. Product and service names mentioned here (OpenAI, Google Gemini, NVIDIA, OpenRouter, Ollama, LM Studio, Windows) belong to their owners; Clickwarden is not affiliated with or endorsed by any of them.

What you can inspect and control

• You choose the provider and accept the data flow. Nothing is sent until you run clickwarden configure, read the notice and answer a confirmation dialog that names the provider and lists what is sent (the default button is No). Acceptance is bound to that host; change the host and you are asked again. clickwarden configure --revoke-consent (or the Revoke consent button) withdraws it. Loopback servers need no acceptance because nothing leaves the machine.

• You always see where your screen goes. The control panel permanently shows Screens → <host> (or Screens stay on this computer) next to a large red STOP button.

• An AI agent cannot do that for you. Choosing a provider, entering the key and accepting data sharing happen only in the settings window, never through the MCP tools.

• Your key is stored encrypted with Windows DPAPI (not in the repo, not in plain text).

• You can see and delete everything it stores, any time. The settings window stays open after Save and has a My data tab listing every stored part (key, settings and consent, session and job records, layouts, saved screenshots, desktop shortcut) with its path, size and a View button (read-only; your key is never shown) and a Delete button; Delete everything wipes it all (default button No). Settings live in %USERPROFILE%\.clickwarden\config, deliberately not in AppData, because packaged desktop apps get a redirected private copy of AppData that other programs cannot see. From a terminal: clickwarden configure --where lists it, clickwarden configure --delete-all deletes it (you type DELETE). Only these known locations are touched; links are never followed.

• Honest identity: requests carry User-Agent: clickwarden/<version>. Clickwarden does not pretend to be another company's client.

• You can read all of it. About 8.7k lines of Python, no binaries, no telemetry. In the clickwarden package the only network client is the vision provider connection in clickwarden/vision.py (git grep "requests" to check); the browser you drive naturally talks to whatever sites you visit. The optional benchmark in benchmarks/ab also calls the Anthropic API, with a key you set yourself.

How it works

agent's text command -> UI Automation control found? -> re-read + hit-test -> safety checks -> SendInput
                                \-> no: capture the window -> vision model (your choice) returns a box
                                        -> fail-closed validation -> safety checks -> SendInput
                     overlay + panel show what it is about to do; the agent gets a short text result

Prefer exposed controls before pixels: fast_controls / fast_click_control read the Windows UI Automation tree and click a control only after re-reading and hit-testing it. Vision is the fallback for things that expose no controls.

Quick start

Requires Windows, Python 3.11+ (developed and tested on 3.14).

git clone https://github.com/kaantokgozsec-hash/clickwarden
cd clickwarden
python -m venv .venv
.\.venv\Scripts\pip install -e .
.\.venv\Scripts\clickwarden configure

clickwarden configure opens a window. Its Set up tab is a checklist (provider, key, data sharing, a safe test, agent connection) and its Connect an agent tab gives copy-paste snippets for Claude Code, Codex and Claude Desktop. On the Provider tab you pick a provider, enter the model and key, read exactly what will be sent, and accept or decline. Paste only the key itself into the key box (the window refuses code samples and huge pastes). If pasting is a problem, clickwarden configure --set-key takes the key in the terminal with hidden input. clickwarden configure --show prints the non-secret settings. Then try:

.\.venv\Scripts\clickwarden doctor
.\.venv\Scripts\clickwarden inspect "What application is in front?"   # one screenshot goes to your provider

Use it from an MCP client

Add a server entry to your client's MCP configuration (the format varies by client):

{
  "mcpServers": {
    "clickwarden": {
      "command": "C:\\path\\to\\clickwarden\\.venv\\Scripts\\python.exe",
      "args": ["C:\\path\\to\\clickwarden\\clickwarden_mcp.py"]
    }
  }
}

The server sends the connecting AI a safety notice and every session start repeats where the screen is sent. Step-by-step setup for Claude Code, Codex and Claude Desktop, the agent skill and a safe demo: docs/CONNECT.md. Tool reference: docs/TOOLS.md; design and rationale: docs/ARCHITECTURE.md; measured results: docs/RESULTS.md. AI agents should also read AGENTS.md. Restart the MCP process after changing settings.

Providers and local models

Any server that accepts OpenAI-style chat/completions with image input can work. The settings window has starting presets (Ollama, LM Studio, OpenRouter, Gemini's OpenAI-compatible endpoint, OpenAI, NVIDIA's API catalog, custom). The maintainer has not tested these presets or any particular model; model names change, so choose a vision-capable model yourself.

Can local vision models run this on their machine? Yes, in principle. Point it at http://localhost:11434/v1/chat/completions (Ollama) or LM Studio's local server, load a vision model, and no screenshot leaves your computer. Practical notes:

• Accuracy is the real limit. Clicking needs precise boxes; small models can be sloppy. Clickwarden validates every answer and refuses to click on absent, ambiguous or malformed results, so a weak model costs time and retries, not stray clicks. Try clickwarden locate "the Start button" first.

• Some models answer on a 0-1000 grid instead of pixels (some Qwen-VL versions do). Set Coordinate format to norm1000 in the settings window if clicks land in the wrong place.

• Slow or busy endpoints (free tiers especially) can take a while to answer. The settings window has a Timeout (default 20 s, allowed 3-120 s); the connection test waits at least 30 s.

• Free and trial endpoints often log what you send and use it to improve their products. Clickwarden warns when a model name ends in :free or the host is NVIDIA's trial API; keep private content off screen.

• Local servers can be slow on large screenshots; the window-crop and Turbo modes help.

• Reasoning models may spend their token budget before answering; keep thinking off (CLICKWARDEN_THINKING=0, default) or raise CLICKWARDEN_LOCATE_MAX_TOKENS.

Built-in safety controls

These reduce risk; they do not remove it. Vision "verification" is a model's opinion, not proof.

Known limits

• One shared Windows cursor: you cannot use the mouse while it works. It is not a background or invisible controller.

• It cannot (and should not) operate the secure desktop, UAC prompts or other users' sessions.

• Applications with elevated privileges may reject input from a normal process.

• Tested mainly on Windows 11 with one developer's setup. Expect rough edges.

• Apps that expose little to UI Automation (games, canvas-drawn apps, some Electron apps) fall back to the vision model, which is slower and less precise. A small vision model can return a slightly wrong box; Clickwarden then refuses to click rather than guess, so the task may stall instead of misclicking.

Benchmark

benchmarks/ab runs the same Claude model on 12 fixed Windows tasks two ways: with Anthropic's computer-use tool (screenshots) and through Clickwarden (text only). It counts both models' tokens from the API's own usage fields and decides success from the end state (files, registry, a local test server), never from the model saying it is done. Results are not published yet; they will be added here with the raw data, failures included.

Repository map

clickwarden/vision.py provider client · providers.py settings, consent, key storage · configure.py settings window · input.py Win32 input and guards · engine.py actions · sessions.py MCP sessions · accessibility.py + uia_probe.ps1 UI Automation reads · grounding.py fail-closed answer validation · server.py MCP tools · tests/ offline suite.

Desktop shortcut with the Clickwarden icon (opens the settings window): python scripts/create_shortcut.py. Check your provider safely: python scripts/provider_check.py shows a neutral test window, sends only that window and compares where the model says the button is with where it really is.

Preview the panel without running anything: python scripts/preview_panel.py.

Tests

python -m unittest discover -s tests -v

The default suite sends no input and makes no network calls. Files named tests/live_*.py are opt-in, drive real windows and may use your provider quota; run them on a quiet desktop.

License and contributing

Apache-2.0. See LICENSE, NOTICE, THIRD_PARTY_NOTICES.md, CONTRIBUTING.md and SECURITY.md. Developed with the assistance of AI coding tools; review the code before trusting it with your desktop.