uitars-mcp

MCP server that gives AI coding agents local GUI grounding — the ability to find any UI element on screen and return its exact pixel coordinates.

Why

Claude Code's built-in computer-use sends every screenshot to the cloud for analysis. This MCP server runs a local vision model instead:

~1.2s per element find (vs cloud round-trip latency)
4.1GB VRAM (runs on any modern GPU)
Fully offline — no API keys, no cloud dependency
90.7% accuracy on ScreenSpot desktop-text benchmark
Native pixel coordinates — returns exact click targets

Setup

1. Download UI-TARS-2B

# Requires ~4.5GB disk space
huggingface-cli download bytedance-research/UI-TARS-2B-SFT --local-dir ./ui-tars-2b

2. Install PyTorch with CUDA

# Install CUDA-enabled PyTorch first (adjust cu126 to your CUDA version)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

3. Install uitars-mcp

pip install uitars-mcp
# or from source:
pip install -e .

4. Configure Claude Code

Add to your Claude Code MCP settings (~/.claude/settings.json):

{
  "mcpServers": {
    "uitars-mcp": {
      "command": "uitars-mcp",
      "env": {
        "UITARS_MODEL": "/path/to/ui-tars-2b"
      }
    }
  }
}

If installed in a venv, use the full path to the executable:

{
  "mcpServers": {
    "uitars-mcp": {
      "command": "/path/to/venv/bin/uitars-mcp",
      "env": {
        "UITARS_MODEL": "/path/to/ui-tars-2b"
      }
    }
  }
}

Tools

Tool	What it does	Latency
`find_element`	Find a UI element by description, returns click coordinates	~1.2s
`describe_screen`	Describe everything visible on screen	~2s
`read_screen_text`	OCR — read all text on screen	~3s
`check_element`	Check element state (enabled, value, etc.)	~1s
`verify_action`	Verify an action worked by checking screen state	~1.5s
`suggest_action`	Suggest next action to achieve a goal	~1.5s
`benchmark`	Measure end-to-end latency	varies

How it works

Takes a screenshot via mss (fast, cross-platform)
Resizes to 1344px wide (optimal vision token count)
Runs UI-TARS-2B inference on GPU
Converts model's 0-1000 normalized coordinates to native screen pixels
Returns coordinates ready for computer-use click tools

The model is lazy-loaded on first call (~3s), then stays in VRAM for subsequent calls.

Environment variables

Variable	Default	Description
`UITARS_MODEL`	(required)	Path to UI-TARS-2B model directory

Requirements

Python 3.10+
NVIDIA GPU with 4.1GB+ VRAM
CUDA-enabled PyTorch
Windows or Linux (macOS untested)