Screen-Aware MCP Server & Antigravity Skill

An open-source Model Context Protocol (MCP) server and Antigravity Skill that provides AI coding assistants with high-fidelity, privacy-first computer screen context, cursor position awareness, active window metadata, and visual verification.

🌟 Overview

When building and debugging graphical interfaces or web applications, verbal references like:

"Make this button blue and increase padding." (while pointing cursor at the button)

require the AI agent to know:

1. What the screen looks like.

2. Exactly where the user is pointing.

3. Which application window and UI component is focused.

Screen-Aware MCP solves this by providing structured, synchronized screen context directly to multimodal AI coding agents (such as Google Antigravity) without relying on permanent disk writes, third-party cloud uploads, or invasive screen-recording daemons.

Related MCP server: Screen Agent

🏛️ Core Architecture

flowchart TD
    User([User + Screen + Mouse])
    User -->|Voice / Text Prompt| Agent[Antigravity Agent]
    Agent -->|Skill Guidance| Skill[Antigravity Skill]
    Skill -->|Tool Invocations| MCP[Screen-Aware MCP Server]
    MCP -->|Win32 / OS APIs| OS[Local OS / Screen / Cursor / Window]
    OS -->|Raw Frames & Coordinates| MCP
    MCP -->|ImageContent + JSON Metadata| Model[Multimodal AI Model]
    Model -->|Visual UI Identification| Agent
    Agent -->|Code Changes & Verification| Codebase[(Source Code)]

Separation of Responsibilities

• Antigravity Skill: Guides the agent on behavior (when to capture screen context, how to correlate cursor coordinates with visual UI elements, how to locate source components, and verify results).

• MCP Server: Provides capabilities (capturing screen frames, measuring cursor coordinates, detecting foreground window, packaging multimodal content).

• OS Platform Layer: Interfaces directly with native OS APIs (GetCursorPos, GetForegroundWindow, mss frame buffer) with full DPI awareness.

• Multimodal Model: Performs visual reasoning over the synchronized screenshot and cursor coordinates.

🚀 Features (Milestone 1 — Read-Only Foundation)

• 📸 High-Speed Screen Capture: Native in-memory screen capture via mss & Pillow.

• 🎯 Sub-Pixel Cursor Tracking: Exact cursor coordinates + normalized [0, 1] positions matching hardware pixels (DPI-aware).

• 🪟 Active Window Detection: Detects focused window title, PID, process executable name, and bounding box.

• ⚡ Synchronized Screen Context: Captures cursor, window, and screen snapshot in a single sub-millisecond call to eliminate coordinate drift.

• 🔒 Privacy-First: Zero disk persistence by default, local execution, no background surveillance.

📦 Project Structure

screen-aware-mcp/
├── server/
│   ├── __init__.py
│   ├── server.py             # MCP Server definition & tool registration
│   ├── config.py             # Server configuration & environment variables
│   ├── tools/
│   │   ├── __init__.py
│   │   ├── screen.py         # capture_screen implementation
│   │   ├── cursor.py         # get_cursor_position implementation
│   │   ├── window.py         # get_active_window implementation
│   │   └── context.py        # capture_screen_context synchronized tool
│   └── platform/
│       ├── __init__.py       # Platform provider selector
│       ├── base.py           # Abstract OS platform interface
│       └── windows.py        # Windows Win32 / ctypes implementation
├── skill/
│   └── SKILL.md              # Antigravity Skill specification
├── voice/
│   └── README.md             # Voice architecture & roadmap
├── tests/
│   ├── test_cursor.py        # Cursor tracking tests
│   ├── test_screen.py        # Screen capture tests
│   ├── test_window.py        # Window detection tests
│   ├── test_context.py       # Context synchronization tests
│   └── test_server.py        # End-to-end MCP tool tests
├── docs/
│   ├── architecture.md       # Detailed architectural design
│   ├── installation.md       # Setup and integration guide
│   ├── antigravity.md        # Antigravity IDE & CLI configuration
│   ├── privacy.md            # Privacy policies & data handling
│   └── security.md           # Security model & tool permissions
├── examples/
│   └── client_example.py     # Example MCP client script
├── .env.example              # Sample environment configuration
├── .gitignore
├── LICENSE                   # MIT License
├── pyproject.toml
└── requirements.txt

🛠️ Installation & Setup

Prerequisites

• Python 3.10 or higher

• Windows 10/11 (macOS / Linux support on roadmap)

1. Install Dependencies

cd screen-aware-mcp
pip install -r requirements.txt

2. Configure Environment (Optional)

cp .env.example .env

3. Run MCP Server

python -m server.server

🧪 Running Tests

Run the full automated test suite:

pytest tests/ -v

🗺️ Roadmap & Milestones

• Milestone 1 — MCP Foundation: Read-only MCP tools (capture_screen, get_cursor_position, get_active_window, capture_screen_context), DPI-aware Win32 integration, comprehensive test suite.

• Milestone 2 — Antigravity Skill: Integrate with Antigravity agent workflows for prompt-driven UI element identification.

• Milestone 3 — Visual UI Targeting: Windows UI Automation (UIA) & accessibility tree integration.

• Milestone 4 — Voice Input: Audio / microphone input bridge.

• Milestone 5 — Controlled UI Interaction: Permission-controlled click, drag, scroll, keyboard input.

• Milestone 6 — Screen Watching: Intelligent frame-diff change detection.

📄 License

This project is licensed under the MIT License.