# Screen-Aware MCP Server & Antigravity Skill
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python: 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/)
[![MCP: 2.x](https://img.shields.io/badge/MCP-2.x-green.svg)](https://modelcontextprotocol.io/)

An open-source Model Context Protocol (MCP) server and Antigravity Skill that provides AI coding assistants with high-fidelity, privacy-first computer screen context, cursor position awareness, active window metadata, and visual verification.

---

## 🌟 Overview

When building and debugging graphical interfaces or web applications, verbal references like:

> *"Make this button blue and increase padding."* (while pointing cursor at the button)

require the AI agent to know:
1. What the screen looks like.
2. Exactly where the user is pointing.
3. Which application window and UI component is focused.

**Screen-Aware MCP** solves this by providing structured, synchronized screen context directly to multimodal AI coding agents (such as Google Antigravity) without relying on permanent disk writes, third-party cloud uploads, or invasive screen-recording daemons.

---

## 🏛️ Core Architecture

```mermaid
flowchart TD
    User([User + Screen + Mouse])
    User -->|Voice / Text Prompt| Agent[Antigravity Agent]
    Agent -->|Skill Guidance| Skill[Antigravity Skill]
    Skill -->|Tool Invocations| MCP[Screen-Aware MCP Server]
    MCP -->|Win32 / OS APIs| OS[Local OS / Screen / Cursor / Window]
    OS -->|Raw Frames & Coordinates| MCP
    MCP -->|ImageContent + JSON Metadata| Model[Multimodal AI Model]
    Model -->|Visual UI Identification| Agent
    Agent -->|Code Changes & Verification| Codebase[(Source Code)]
```

### Separation of Responsibilities

* **Antigravity Skill**: Guides the agent on *behavior* (when to capture screen context, how to correlate cursor coordinates with visual UI elements, how to locate source components, and verify results).
* **MCP Server**: Provides *capabilities* (capturing screen frames, measuring cursor coordinates, detecting foreground window, packaging multimodal content).
* **OS Platform Layer**: Interfaces directly with native OS APIs (`GetCursorPos`, `GetForegroundWindow`, `mss` frame buffer) with full DPI awareness.
* **Multimodal Model**: Performs visual reasoning over the synchronized screenshot and cursor coordinates.

---

## 🚀 Features (Milestone 1 — Read-Only Foundation)

- 📸 **High-Speed Screen Capture**: Native in-memory screen capture via `mss` & Pillow.
- 🎯 **Sub-Pixel Cursor Tracking**: Exact cursor coordinates + normalized `[0, 1]` positions matching hardware pixels (DPI-aware).
- 🪟 **Active Window Detection**: Detects focused window title, PID, process executable name, and bounding box.
- ⚡ **Synchronized Screen Context**: Captures cursor, window, and screen snapshot in a single sub-millisecond call to eliminate coordinate drift.
- 🔒 **Privacy-First**: Zero disk persistence by default, local execution, no background surveillance.

---

## 📦 Project Structure

```
screen-aware-mcp/
├── server/
│   ├── __init__.py
│   ├── server.py             # MCP Server definition & tool registration
│   ├── config.py             # Server configuration & environment variables
│   ├── tools/
│   │   ├── __init__.py
│   │   ├── screen.py         # capture_screen implementation
│   │   ├── cursor.py         # get_cursor_position implementation
│   │   ├── window.py         # get_active_window implementation
│   │   └── context.py        # capture_screen_context synchronized tool
│   └── platform/
│       ├── __init__.py       # Platform provider selector
│       ├── base.py           # Abstract OS platform interface
│       └── windows.py        # Windows Win32 / ctypes implementation
├── skill/
│   └── SKILL.md              # Antigravity Skill specification
├── voice/
│   └── README.md             # Voice architecture & roadmap
├── tests/
│   ├── test_cursor.py        # Cursor tracking tests
│   ├── test_screen.py        # Screen capture tests
│   ├── test_window.py        # Window detection tests
│   ├── test_context.py       # Context synchronization tests
│   └── test_server.py        # End-to-end MCP tool tests
├── docs/
│   ├── architecture.md       # Detailed architectural design
│   ├── installation.md       # Setup and integration guide
│   ├── antigravity.md        # Antigravity IDE & CLI configuration
│   ├── privacy.md            # Privacy policies & data handling
│   └── security.md           # Security model & tool permissions
├── examples/
│   └── client_example.py     # Example MCP client script
├── .env.example              # Sample environment configuration
├── .gitignore
├── LICENSE                   # MIT License
├── pyproject.toml
└── requirements.txt
```

---

## 🛠️ Installation & Setup

### Prerequisites
- Python 3.10 or higher
- Windows 10/11 (macOS / Linux support on roadmap)

### 1. Install Dependencies
```bash
cd screen-aware-mcp
pip install -r requirements.txt
```

### 2. Configure Environment (Optional)
```bash
cp .env.example .env
```

### 3. Run MCP Server
```bash
python -m server.server
```

---

## 🧪 Running Tests

Run the full automated test suite:
```bash
pytest tests/ -v
```

---

## 🗺️ Roadmap & Milestones

- [x] **Milestone 1 — MCP Foundation**: Read-only MCP tools (`capture_screen`, `get_cursor_position`, `get_active_window`, `capture_screen_context`), DPI-aware Win32 integration, comprehensive test suite.
- [ ] **Milestone 2 — Antigravity Skill**: Integrate with Antigravity agent workflows for prompt-driven UI element identification.
- [ ] **Milestone 3 — Visual UI Targeting**: Windows UI Automation (UIA) & accessibility tree integration.
- [ ] **Milestone 4 — Voice Input**: Audio / microphone input bridge.
- [ ] **Milestone 5 — Controlled UI Interaction**: Permission-controlled click, drag, scroll, keyboard input.
- [ ] **Milestone 6 — Screen Watching**: Intelligent frame-diff change detection.

---

## 📄 License

This project is licensed under the [MIT License](LICENSE).