# uitars-mcp

MCP server that gives AI coding agents local GUI grounding — the ability to find any UI element on screen and return its exact pixel coordinates.

Powered by [UI-TARS-2B](https://huggingface.co/bytedance-research/UI-TARS-2B-SFT), ByteDance's 2B parameter GUI grounding model.

## Why

Claude Code's built-in computer-use sends every screenshot to the cloud for analysis. This MCP server runs a local vision model instead:

- **~1.2s per element find** (vs cloud round-trip latency)
- **4.1GB VRAM** (runs on any modern GPU)
- **Fully offline** — no API keys, no cloud dependency
- **90.7% accuracy** on ScreenSpot desktop-text benchmark
- **Native pixel coordinates** — returns exact click targets

## Setup

### 1. Download UI-TARS-2B

```bash
# Requires ~4.5GB disk space
huggingface-cli download bytedance-research/UI-TARS-2B-SFT --local-dir ./ui-tars-2b
```

### 2. Install PyTorch with CUDA

```bash
# Install CUDA-enabled PyTorch first (adjust cu126 to your CUDA version)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
```

### 3. Install uitars-mcp

```bash
pip install uitars-mcp
# or from source:
pip install -e .
```

### 4. Configure Claude Code

Add to your Claude Code MCP settings (`~/.claude/settings.json`):

```json
{
  "mcpServers": {
    "uitars-mcp": {
      "command": "uitars-mcp",
      "env": {
        "UITARS_MODEL": "/path/to/ui-tars-2b"
      }
    }
  }
}
```

If installed in a venv, use the full path to the executable:

```json
{
  "mcpServers": {
    "uitars-mcp": {
      "command": "/path/to/venv/bin/uitars-mcp",
      "env": {
        "UITARS_MODEL": "/path/to/ui-tars-2b"
      }
    }
  }
}
```

## Tools

| Tool | What it does | Latency |
|------|-------------|---------|
| `find_element` | Find a UI element by description, returns click coordinates | ~1.2s |
| `describe_screen` | Describe everything visible on screen | ~2s |
| `read_screen_text` | OCR — read all text on screen | ~3s |
| `check_element` | Check element state (enabled, value, etc.) | ~1s |
| `verify_action` | Verify an action worked by checking screen state | ~1.5s |
| `suggest_action` | Suggest next action to achieve a goal | ~1.5s |
| `benchmark` | Measure end-to-end latency | varies |

## How it works

1. Takes a screenshot via `mss` (fast, cross-platform)
2. Resizes to 1344px wide (optimal vision token count)
3. Runs UI-TARS-2B inference on GPU
4. Converts model's 0-1000 normalized coordinates to native screen pixels
5. Returns coordinates ready for `computer-use` click tools

The model is lazy-loaded on first call (~3s), then stays in VRAM for subsequent calls.

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `UITARS_MODEL` | (required) | Path to UI-TARS-2B model directory |

## Requirements

- Python 3.10+
- NVIDIA GPU with 4.1GB+ VRAM
- CUDA-enabled PyTorch
- Windows or Linux (macOS untested)