VNC MCP Server

An advanced Model Context Protocol (MCP) server that empowers AI agents to see, reason, and control computers via the VNC (RFB) protocol. This server transforms a standard VNC connection into a high-level "Vision-Action" interface for AI Agents.

📊 System Architecture

graph TD
    subgraph "AI Environment"
        A[AI Agent / Claude Desktop] -->|MCP Protocol| B[VNC MCP Server]
    end

    subgraph "Bridge Layer"
        B -->|Image Processing| C[Sharp / Tesseract.js]
        B -->|RFB Protocol| D[VNC Connection]
    end

    subgraph "Target System"
        D -->|Input Simulation| E[Windows / macOS / Linux]
        E -->|Framebuffer| D
    end

    style B fill:#f00,stroke:#333,stroke-width:4px

Related MCP server: Wayland MCP Server

🌟 Key Features

👁️ Advanced Vision

• Regional Screenshot: Precise screen capture of specific areas to save tokens.

• OCR (Optical Character Recognition): Extract text from any part of the screen using Tesseract.js.

• Visual Search: Find specific icons, buttons, or images on the screen using template matching.

• Pixel Analysis: Get exact Hex/RGB colors of any pixel for state verification.

• Change Detection: Wait for a specific region to update before proceeding.

🕹️ Human-Like Control

• Precise Input: Single clicks, double clicks, and smooth mouse movement.

• Drag & Drop: Simulate complex dragging operations between coordinates.

• Scrolling: Directional mouse wheel support (Up/Down/Left/Right).

• Keyboard Mastery: Type strings and perform complex key combinations (e.g., Ctrl+C, Cmd+Space).

⚡ Hybrid Control (VNC + SSH Shell)

• Background Execution: Run any shell command (bash/cmd/powershell) on the server.

• 🔗 Remote Execution via SSH: Run commands directly on the VNC host via integrated SSH client.

• 🛡️ Security Layer: Robust filtering system with platform-specific allowlists and denylists.

• Cross-Platform Support: Works on Windows, Linux, and macOS.

• Seamless Integration: Use terminal to install software then use VNC to control apps.

🧠 Intelligence & Health

• OS Detection & Platform Awareness: Intelligent OS detection based on screen resolution patterns (Windows/macOS/Linux).

• Connection Health: Real-time connectivity monitoring with latency measurement and heartbeat status.

• Clipboard Sync: Full bidirectional clipboard support.

• Auto-Connect: Configure server to connect immediately via environment variables.

• Advanced Sequencing: Execute multiple actions in a single call for faster automation.

🚀 Getting Started

Prerequisites

• Node.js v18+

• A running VNC server on the target system.

• (Optional) SSH service enabled on the target system for remote command execution.

Quick Start

1. Clone & Install:

   git clone https://github.com/volkan-m/vnc-mcp-server.git
   cd vnc-mcp-server
   npm install
   npm run build

2. Test Environment (Docker with Chrome & SSH): Run a full Linux desktop with Google Chrome, VNC, and SSH enabled:

   docker run -d \
     -p 5901:5900 \
     -p 2222:22 \
     -v /dev/shm:/dev/shm \
     --name mcp-vnc-chrome \
     -e VNC_PASSWORD=vncpass \
     dorowu/ubuntu-desktop-lxde-vnc:latest

   Connect to VNC at localhost:5901 and SSH at localhost:2222 (port 22 inside).

3. MCP Client Configuration:

Google Antigravity / Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "vnc": {
      "command": "node",
      "args": ["/absolute/path/to/vnc-mcp-server/dist/index.js"],
      "env": {
        "VNC_HOST": "localhost",
        "VNC_PORT": "5901",
        "VNC_PASSWORD": "vncpass"
      }
    }
  }
}

🛡️ Security Configuration (Optional)

You can restrict which system commands the AI can execute:

🔗 Remote Execution via SSH

The server now supports direct command execution on the target machine via SSH. When connecting through vnc_connect, you can provide an optional ssh object:

{
  "host": "192.168.1.50",
  "ssh": {
    "user": "root",
    "password": "your-password",
    "port": 22
  }
}

If SSH is connected, vnc_system_command automatically routes all commands to the target machine instead of the local server.

📊 Enhanced Status Monitoring

Screen Info

vnc_get_screen_info returns detailed information about the remote session:

• Resolution: Current display dimensions

• OS Hint: Intelligent OS detection (Windows/macOS/Linux based on resolution patterns)

• SSH Status: Whether SSH connection is active

Connection Health

vnc_get_connection_health provides comprehensive connectivity metrics:

• VNC Status: Connected/Disconnected state

• Latency: Real-time latency measurement from screen refresh operations

• Resolution: Current display dimensions

• SSH Status: SSH connection state

🎯 Advanced Vision Features

Screenshot with Coordinate Overlay

Use the overlay parameter in vnc_screenshot to add a grid with labeled coordinates (A1, B2, etc.) for easier navigation:

{
  "format": "jpeg",
  "quality": 85,
  "overlay": true
}

This helps AI agents understand and communicate screen positions more accurately.

OCR with Multiple Languages

The vnc_ocr_region tool supports multiple languages via Tesseract.js. Specify the language code:

{
  "x": 0,
  "y": 0,
  "width": 1920,
  "height": 1080,
  "lang": "eng+deu+fra"
}

Supported languages: eng (English), deu (German), fra (French), spa (Spanish), and many more.

Batch Actions with vnc_execute_sequence

Execute multiple input actions in a single call for faster automation:

{
  "actions": [
    {"type": "move", "x": 100, "y": 100},
    {"type": "click", "x": 100, "y": 100},
    {"type": "wait", "ms": 500},
    {"type": "type", "text": "Hello World"},
    {"type": "key", "key": "Enter"}
  ]
}

This approach reduces RPC overhead and improves responsiveness for complex automation sequences.

🧰 API Reference (Tools)

🛡 License

MIT License. See LICENSE for details.