uitars-mcp

MCP server that gives AI coding agents local GUI grounding — the ability to find any UI element on screen and return its exact pixel coordinates.

Powered by UI-TARS-2B, ByteDance's 2B parameter GUI grounding model.

Why

Claude Code's built-in computer-use sends every screenshot to the cloud for analysis. This MCP server runs a local vision model instead:

• ~1.2s per element find (vs cloud round-trip latency)

• 4.1GB VRAM (runs on any modern GPU)

• Fully offline — no API keys, no cloud dependency

• 90.7% accuracy on ScreenSpot desktop-text benchmark

• Native pixel coordinates — returns exact click targets

Setup

1. Download UI-TARS-2B

# Requires ~4.5GB disk space
huggingface-cli download bytedance-research/UI-TARS-2B-SFT --local-dir ./ui-tars-2b

2. Install PyTorch with CUDA

# Install CUDA-enabled PyTorch first (adjust cu126 to your CUDA version)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

3. Install uitars-mcp

pip install uitars-mcp
# or from source:
pip install -e .

4. Configure Claude Code

Add to your Claude Code MCP settings (~/.claude/settings.json):

{
  "mcpServers": {
    "uitars-mcp": {
      "command": "uitars-mcp",
      "env": {
        "UITARS_MODEL": "/path/to/ui-tars-2b"
      }
    }
  }
}

If installed in a venv, use the full path to the executable:

{
  "mcpServers": {
    "uitars-mcp": {
      "command": "/path/to/venv/bin/uitars-mcp",
      "env": {
        "UITARS_MODEL": "/path/to/ui-tars-2b"
      }
    }
  }
}

Tools

How it works

1. Takes a screenshot via mss (fast, cross-platform)

2. Resizes to 1344px wide (optimal vision token count)

3. Runs UI-TARS-2B inference on GPU

4. Converts model's 0-1000 normalized coordinates to native screen pixels

5. Returns coordinates ready for computer-use click tools

The model is lazy-loaded on first call (~3s), then stays in VRAM for subsequent calls.

Environment variables

Requirements

• Python 3.10+

• NVIDIA GPU with 4.1GB+ VRAM

• CUDA-enabled PyTorch

• Windows or Linux (macOS untested)