# VNC MCP Server

An advanced Model Context Protocol (MCP) server that empowers AI agents to see, reason, and control computers via the VNC (RFB) protocol. This server transforms a standard VNC connection into a high-level "Vision-Action" interface for AI Agents.

## 📊 System Architecture

```mermaid
graph TD
    subgraph "AI Environment"
        A[AI Agent / Claude Desktop] -->|MCP Protocol| B[VNC MCP Server]
    end

    subgraph "Bridge Layer"
        B -->|Image Processing| C[Sharp / Tesseract.js]
        B -->|RFB Protocol| D[VNC Connection]
    end

    subgraph "Target System"
        D -->|Input Simulation| E[Windows / macOS / Linux]
        E -->|Framebuffer| D
    end

    style B fill:#f00,stroke:#333,stroke-width:4px
```

## 🌟 Key Features

### 👁️ Advanced Vision
- **Regional Screenshot**: Precise screen capture of specific areas to save tokens.
- **OCR (Optical Character Recognition)**: Extract text from any part of the screen using Tesseract.js.
- **Visual Search**: Find specific icons, buttons, or images on the screen using template matching.
- **Pixel Analysis**: Get exact Hex/RGB colors of any pixel for state verification.
- **Change Detection**: Wait for a specific region to update before proceeding.

### 🕹️ Human-Like Control
- **Precise Input**: Single clicks, **double clicks**, and smooth mouse movement.
- **Drag & Drop**: Simulate complex dragging operations between coordinates.
- **Scrolling**: Directional mouse wheel support (Up/Down/Left/Right).
- **Keyboard Mastery**: Type strings and perform complex key combinations (e.g., `Ctrl+C`, `Cmd+Space`).

### ⚡ Hybrid Control (VNC + SSH Shell)
- **Background Execution**: Run any shell command (bash/cmd/powershell) on the server.
- **🔗 Remote Execution via SSH**: Run commands directly on the VNC host via integrated SSH client.
- **🛡️ Security Layer**: Robust filtering system with platform-specific allowlists and denylists.
- **Cross-Platform Support**: Works on Windows, Linux, and macOS.
- **Seamless Integration**: Use terminal to install software then use VNC to control apps.

### 🧠 Intelligence & Health
- **OS Detection & Platform Awareness**: Intelligent OS detection based on screen resolution patterns (Windows/macOS/Linux).
- **Connection Health**: Real-time connectivity monitoring with latency measurement and heartbeat status.
- **Clipboard Sync**: Full bidirectional clipboard support.
- **Auto-Connect**: Configure server to connect immediately via environment variables.
- **Advanced Sequencing**: Execute multiple actions in a single call for faster automation.

## 🚀 Getting Started

### Prerequisites
- Node.js v18+
- A running VNC server on the target system.
- (Optional) SSH service enabled on the target system for remote command execution.

### Quick Start
1. Clone & Install:
   ```bash
   git clone https://github.com/volkan-m/vnc-mcp-server.git
   cd vnc-mcp-server
   npm install
   npm run build
   ```

2. **Test Environment (Docker with Chrome & SSH):**
   Run a full Linux desktop with Google Chrome, VNC, and SSH enabled:
   ```bash
   docker run -d \
     -p 5901:5900 \
     -p 2222:22 \
     -v /dev/shm:/dev/shm \
     --name mcp-vnc-chrome \
     -e VNC_PASSWORD=vncpass \
     dorowu/ubuntu-desktop-lxde-vnc:latest
   ```
   *Connect to VNC at localhost:5901 and SSH at localhost:2222 (port 22 inside).*

3. **MCP Client Configuration:**

#### Google Antigravity / Claude Desktop
Add this to your `claude_desktop_config.json`:
```json
{
  "mcpServers": {
    "vnc": {
      "command": "node",
      "args": ["/absolute/path/to/vnc-mcp-server/dist/index.js"],
      "env": {
        "VNC_HOST": "localhost",
        "VNC_PORT": "5901",
        "VNC_PASSWORD": "vncpass"
      }
    }
  }
}
```

#### 🛡️ Security Configuration (Optional)
You can restrict which system commands the AI can execute:

| Variable | Description | Default Denied |
|---|---|---|
| `LINUX_DENIED_COMMANDS` | Blocked Linux commands | `rm -rf,rm -f,shred,mkfs,:(){ :|:& };:` |
| `WIN_DENIED_COMMANDS` | Blocked Windows commands | `del,format,rd,sfc,attrib` |
| `MAC_DENIED_COMMANDS` | Blocked macOS commands | `rm -rf,rm -f` |
| `[OS]_ALLOWED_COMMANDS` | If set, **ONLY** these are allowed | (Empty by default) |

## 🔗 Remote Execution via SSH
The server now supports direct command execution on the target machine via SSH. When connecting through `vnc_connect`, you can provide an optional `ssh` object:

```json
{
  "host": "192.168.1.50",
  "ssh": {
    "user": "root",
    "password": "your-password",
    "port": 22
  }
}
```
If SSH is connected, `vnc_system_command` automatically routes all commands to the target machine instead of the local server.

## 📊 Enhanced Status Monitoring

### Screen Info
`vnc_get_screen_info` returns detailed information about the remote session:
- **Resolution**: Current display dimensions
- **OS Hint**: Intelligent OS detection (Windows/macOS/Linux based on resolution patterns)
- **SSH Status**: Whether SSH connection is active

### Connection Health  
`vnc_get_connection_health` provides comprehensive connectivity metrics:
- **VNC Status**: Connected/Disconnected state
- **Latency**: Real-time latency measurement from screen refresh operations
- **Resolution**: Current display dimensions
- **SSH Status**: SSH connection state

## 🎯 Advanced Vision Features

### Screenshot with Coordinate Overlay
Use the `overlay` parameter in `vnc_screenshot` to add a grid with labeled coordinates (A1, B2, etc.) for easier navigation:

```json
{
  "format": "jpeg",
  "quality": 85,
  "overlay": true
}
```

This helps AI agents understand and communicate screen positions more accurately.

### OCR with Multiple Languages
The `vnc_ocr_region` tool supports multiple languages via Tesseract.js. Specify the language code:

```json
{
  "x": 0,
  "y": 0,
  "width": 1920,
  "height": 1080,
  "lang": "eng+deu+fra"
}
```

Supported languages: eng (English), deu (German), fra (French), spa (Spanish), and many more.

### Batch Actions with `vnc_execute_sequence`
Execute multiple input actions in a single call for faster automation:

```json
{
  "actions": [
    {"type": "move", "x": 100, "y": 100},
    {"type": "click", "x": 100, "y": 100},
    {"type": "wait", "ms": 500},
    {"type": "type", "text": "Hello World"},
    {"type": "key", "key": "Enter"}
  ]
}
```

This approach reduces RPC overhead and improves responsiveness for complex automation sequences.

## 🧰 API Reference (Tools)

| Tool | Category | Description |
|---|---|---|
| `vnc_connect` | Setup | Connect to VNC (optional: include `ssh` configuration). |
| `vnc_disconnect` | Setup | Disconnect from VNC server. |
| `vnc_get_screen_info` | Status | Returns resolution, OS hint (Windows/macOS/Linux), and SSH status. |
| `vnc_get_connection_health`| Status | Provides heartbeat status, latency measurement, resolution, and SSH connection state. |
| `vnc_screenshot` | Vision | Captures PNG/JPEG of full screen or region with optional coordinate overlay. |
| `vnc_ocr_region` | Vision | Extracts text from a specific region with OCR (supports multiple languages). |
| `vnc_find_image` | Vision | Finds a template image on screen with configurable tolerance. |
| `vnc_wait_for_change` | Vision | Blocks until a screen region updates within timeout. |
| `vnc_get_pixel_color` | Vision | Returns Hex/RGB color value of a specific pixel. |
| `vnc_mouse_move` | Input | Moves mouse to coordinates. |
| `vnc_mouse_click` | Input | Performs mouse clicks (Left/Middle/Right buttons). |
| `vnc_mouse_double_click` | Input | Performs a quick double click. |
| `vnc_mouse_drag` | Input | Drags from point A to point B with button control. |
| `vnc_mouse_scroll` | Input | Directional wheel scrolling (Up/Down/Left/Right). |
| `vnc_key_tap` | Input | Taps keys with modifiers (e.g., `C-v`, `M-Tab`, `S-Enter`). |
| `vnc_type_string` | Input | Types a sequence of characters. |
| `vnc_execute_sequence` | Input | Executes multiple input actions in a single call for performance. |
| `vnc_system_command` | System | **Execute shell commands (Local or Remote via SSH).** |
| `vnc_launch_gui_app` | System | Launches GUI applications on the remote desktop. |
| `vnc_set_clipboard` | System | Sets remote system clipboard text. |
| `vnc_get_clipboard` | System | Retrieves last known remote clipboard content. |

## 🛡 License
MIT License. See [LICENSE](LICENSE) for details.