Extra
by AIYantra
README.md
<div align="center">
<a href="https://extra.yantraos.com">
<img src="assets/logo.png" alt="Extra — Get extra from your AI" width="420" />
</a>
# Extra
### The Open-Source Astra 6 for your Linux PC, Mac & Windows
**Your AI can see, click, type, navigate, and get real work done on Linux, macOS & Windows.**
<!-- mcp-name: io.github.AIYantra/extra -->
[](https://pepy.tech/projects/extra-desktop)
[](LICENSE)
[](https://extra.yantraos.com)
[](https://modelcontextprotocol.io)
[](https://yantraos.com)
[](https://github.com/AIYantra/extra)
[Website](https://extra.yantraos.com) • [Quickstart](#-quickstart-1-minute) • [Linux Guide](README_LINUX.md) • [macOS Guide](README_MACOS.md) • [Architecture](docs/ARCHITECTURE.md) • [License](LICENSE)
<br/>
<br/>
<a href="https://extra.yantraos.com">
<img src="https://raw.githubusercontent.com/AIYantra/extra/main/assets/demo.gif" alt="Extra in action — Live Windows desktop automation" width="100%" />
</a>
</div>
---
## What is Extra and Why Does It Exist?
Extra is an open-source Computer Use protocol and Model Context Protocol (MCP) server developed by yantraOS. It bridges LLMs (including ChatGPT, Claude, Cursor, and Gemini) directly to physical desktop operating systems across Linux, macOS, and Windows. Extra replaces fragile vision-only screenshot loops with native OS accessibility trees and sub-3ms GPU frame streaming, delivering pixel-perfect clicking, instant typing, and up to 225× lower token costs.
Today's AI assistants can write brilliant essays and code, but they are trapped inside a chat window. When you need them to click a button, open an app, organize your files, or input data into a spreadsheet, they can only give you text instructions.
**Extra gives your AI hands and eyes on Linux, macOS & Windows.**
It connects **Claude, Google Antigravity, Cursor, AGY**, or any autonomous agent directly to your desktop with high-speed screen vision, pixel-perfect clicking, and instant typing.
* **No clunky browser extensions.**
* **No expensive cloud servers watching your screen.**
* **100% open source, local, and private.**
---
## What Can an AI Assistant Do With Extra?
Once Extra is running, you can talk to your AI like a real human assistant sitting at your desk:
* 📊 **Spreadsheets & Data:** *"Open Excel or LibreOffice Calc, calculate total revenue from the invoice CSVs in my Downloads folder, and create a summary chart."*
* 🗂️ **Desktop & File Cleanup:** *"Clean up my messy desktop by moving screenshots into Pictures, PDFs into Documents, and deleting empty folders."*
* 🌐 **Web Research & Data Entry:** *"Open Chrome or Firefox, find the top 5 flights to Tokyo under $800, and copy their flight numbers and dates into a text note."*
* ⚙️ **System Administration:** *"Open Settings or terminal, check system updates, and verify service health."*
* 🎵 **App Control:** *"Launch Spotify or VLC, search for low-fi focus beats, and start playing."*
---
## ⚡ Quickstart (1 Minute)
You can set up Extra in seconds on **Linux**, **macOS**, or **Windows**:
### Option A: Ask Your AI To Set It Up (Primary Funnel — Easiest)
Copy and paste the single directive for your OS into **Claude Desktop**, **Google Antigravity**, **Cursor**, **Windsurf**, or any MCP agent:
* **🐧 For Linux (Arch / yantraOS, Ubuntu, Debian, Fedora):**
```text
Setup Extra on my Linux machine: In Terminal run 'curl -sSL https://extra.yantraos.com/install.sh | bash', then read and configure ~/.extra/app/STARTER_PROMPT_LINUX.md so we are ready to use Extra.
```
* **🍏 For macOS (Apple Silicon M1-M4 & Intel):**
```text
Setup Extra on my Mac: In Terminal run 'curl -sSL https://extra.yantraos.com/install.sh | bash', then read and configure ~/.extra/app/STARTER_PROMPT_MACOS.md so we are ready to use Extra.
```
* **🪟 For Windows (11 / 10):**
```text
Setup Extra on my PC: In PowerShell run 'iwr -useb https://extra.yantraos.com/install.ps1 -OutFile "$env:TEMP\install_extra.ps1"; & "$env:TEMP\install_extra.ps1"', then read and configure ~/.extra/app/STARTER_PROMPT.md so we are ready to use Extra.
```
Your AI will run the installer, configure MCP and local rules, and reply:
> **"We are ready! Please restart <your AI application, e.g. Claude Desktop, Antigravity, Cursor, Windsurf> to make it work."**
---
### Option B: Run the One-Liner Yourself
* **🐧 Linux Terminal (Arch / yantraOS, Ubuntu, Debian, Fedora):**
```bash
curl -sSL https://extra.yantraos.com/install.sh | bash
```
* **🍏 macOS Terminal:**
```bash
curl -sSL https://extra.yantraos.com/install.sh | bash
```
* **🪟 Windows PowerShell (Staged, Defender-Clean):**
```powershell
iwr -useb https://extra.yantraos.com/install.ps1 -OutFile "$env:TEMP\install_extra.ps1"; & "$env:TEMP\install_extra.ps1"
```
The automated installer will:
* Verify OS architecture (Linux Wayland/X11, macOS 12.3+, or Windows 10/11 64-bit).
* Provision isolated Python 3.10+ runtime (`~/.extra/venv`).
* Auto-configure **Claude Desktop**, **Cursor**, **Windsurf**, and **Antigravity** (`agy`).
* Run the health diagnostic doctor and present the 1-step starter prompt.
---
### Option C: Developer Git Clone
```bash
# Linux / macOS:
git clone https://github.com/AIYantra/extra.git ~/.extra/app
cd ~/.extra/app && ./install.sh
# Windows:
git clone https://github.com/AIYantra/extra.git "$HOME\.extra\app"
cd "$HOME\.extra\app" ; .\install.ps1
```
---
### Connecting to Any MCP-Compatible AI
Extra works out of the box with any agent supporting the **Model Context Protocol (MCP)**. If you use Cursor, Windsurf, or custom agents, add this snippet to your MCP config:
```json
{
"mcpServers": {
"extra": {
"command": "npx",
"args": ["-y", "@yantraos/extra-desktop"]
}
}
}
```
Or using native Python:
```json
{
"mcpServers": {
"extra": {
"command": "python",
"args": ["-m", "extra.mcp.server"]
}
}
}
```
---
## How Does Extra Work Under the Hood?
Extra operates via a sovereign Dual-Plane Architecture that decouples visual perception from hardware actuation. For perception, Extra combines sub-3ms GPU frame capture (DirectX DXGI, Metal, Spectacle) with OS accessibility hierarchy inspection (COM UI Automation, AT-SPI2, Quartz). For actuation, Extra dispatches lossless kernel and Win32 hardware events with zero coordinate drift, guaranteeing deterministic control across all resolutions and display scaling factors.
Extra is engineered from the ground up for speed, reliability, and token efficiency:
1. 👁️ **Ultra-Fast Screen Capture (< 3ms):** Uses native DirectX Desktop Duplication (DXGI) to take crystal-clear desktop frames in under 3 milliseconds—without lagging your PC or blurring text.
2. 🎯 **Pixel-Perfect Clicking:** Rather than guessing coordinates from fuzzy screenshots, Extra queries the native Windows accessibility tree (UI Automation) to click the exact button, menu, or text field with 100% mathematical accuracy.
3. ⚡ **Instant Typing:** Types 500 characters in under 5 milliseconds via native Win32 Unicode injection—with zero dropped letters and full support for emojis and international languages.
4. 🛑 **Infinite Loop Stall Protection:** If an app freezes or a click produces no visual result, Extra immediately catches it and stops safely instead of burning your tokens in an endless loop.
5. 🔒 **100% Local & Private:** Extra runs completely on your machine. Zero screenshots, keystrokes, or telemetry are ever sent to any cloud server.
---
## 📊 Performance Comparison
| Metric / Capability | Legacy Scripts (`PyAutoGUI`) | Cloud Vision Models | **Extra Engine** |
| :--- | :--- | :--- | :--- |
| **Screen Grab Speed** | 150 – 300 ms | 80 – 150 ms | **1.8 – 3.2 ms (DirectX DXGI)** |
| **Typing Speed (100 chars)** | 2.5 – 5.0 seconds | 1.0 – 2.0 seconds | **< 0.005 seconds (Instant Win32)** |
| **Click Accuracy** | ~60% (fails on display scaling) | ~82% (vision guess) | **99.4% (Native Windows UIA)** |
| **Multi-Monitor DPI Support** | Broken on 125%/150% scales | Requires manual adjustment | **Automatic (PerMonitorV2)** |
| **AI Token Cost** | High (full screenshot every step) | High (full vision payload) | **70% Lower (Smart element tree)** |
| **Stall Prevention** | None (gets stuck forever) | Basic timeout | **Smart visual delta detection** |
---
## 🛠️ Included Tools (MCP Suite — 20 Native Tools)
When connected to Extra, your AI assistant receives full sovereign desktop control across 20 native tools:
| Tool Name | What It Does |
| :--- | :--- |
| `extra_launch` | Opens any app, utility, or URL directly (`calc`, `notepad`, `settings`, `edge`, `chrome`) |
| `extra_focus_window` | Brings any window to the front via native thread input attachment |
| `extra_click` | Clicks with sub-pixel DPI accuracy, human-like Bezier curves, and semantic visual grounding (`target="..."`) |
| `extra_stroke` | Draws fluid, continuous brush splines across waypoints with pressure and Catmull-Rom smoothing |
| `extra_type` | Injects text instantly with zero lag, full emoji support, and Win32 Unicode packets |
| `extra_hotkey` | Sends keyboard shortcuts (`Ctrl+C`, `Win+E`, `Alt+Tab`, `Enter`) |
| `extra_batch_actions` | Executes atomic lists of hardware & SOUL reflex actions (`eval`, `assert`, `wait_for_state`) in sub-milliseconds |
| `extra_snap_layout` | Programmatically docks and arranges windows in side-by-side or split layouts in < 15ms |
| `extra_fs_batch` | High-speed batch filesystem operations compliant with Windows Defender Controlled Folder Access |
| `extra_inspect_ui` | Scans accessible UI trees and interactive element nodes in real time |
| `extra_click_element` | Clicks UI elements deterministically by accessible ID or semantic target |
| `extra_scroll` | Smoothly scrolls wheels up, down, left, or right |
| `extra_drag` | Drags and drops files, windows, or sliders between coordinates |
| `extra_screenshot` | Captures high-res desktop frames in < 25ms with optional Set-of-Mark visual badges |
| `extra_browser` | Directly extracts DOM content in Edge/Chrome or triggers synthesized API fast-paths |
| `extra_task_start` | Activates ambient screen edge pulse and cursor tracking halo |
| `extra_task_complete` | Signals task completion: flashes emerald border, plays acoustic chime, and analyzes trajectory |
| `extra_indicate_status` | Direct programmatic control over active, complete, and idle desktop indicators |
| `extra_recall_memory` | Recalls past task workflows, artifacts, and known quirks via KùzuDB + FastEmbed in < 2ms |
| `extra_scout_app` | Discovers UI frameworks (Electron/Win32/Viewport), universal hotkeys, and generates SKILL playbooks |
| `extra_evolve_skill` | Crystallizes newly verified zero-stall fast paths into permanent skill playbooks |
---
## 🏛️ Repository Structure
```text
extra/
├── assets/ # Brand logos and banners
├── bin/ # Node.js CLI executable (npx @yantraos/extra-desktop)
├── core/ # Sovereign Core Automation Engine
│ ├── platform/ # Platform Abstraction Layer (PAL facade for Win32 & macOS)
│ │ ├── windows/ # Win32 SendInput, DXGI capture, UIA, and focus
│ │ └── macos/ # ScreenCaptureKit, CoreGraphics, and AXUIElement
│ ├── soul/ # System One Ultra-fast Layer (Decider, Eyes, Grammar, Schemas)
│ ├── memory/ # KùzuDB episodic knowledge graph & FastEmbed embeddings
│ ├── evolution/ # Autonomous skill crystallizer & API synthesizer
│ ├── scout/ # Application discovery, CDP sniffer, and session vault
│ ├── motion/ # Human-like Bezier cursor flight and spline dynamics
│ ├── capture.py # Unified high-speed desktop screen capture
│ ├── geometry.py # PerMonitorV2 DPI scaling & display normalization
│ ├── indicators.py # Ambient screen pulse, cursor halo, and harmonic chime
│ ├── input_engine.py # PAL input facade (clicks, strokes, typing, batching)
│ ├── focus.py # Window activation, snapping, and layout management
│ ├── uia_plane.py # Accessibility tree inspection and Set-of-Mark labeling
│ └── stall_breaker.py # Closed-loop perceptual diffing & safety killswitch
├── fastpath/ # High-speed deterministic execution
│ ├── shell.py # Native app launcher with crash-recovery auto-suppression
│ ├── browser.py # Playwright & Chromium DevTools Protocol (CDP) bridge
│ ├── fs.py # CFA-compliant batch filesystem engine
│ └── web/ # Synthesized Web-to-API fast-paths
├── mcp/ # Model Context Protocol
│ └── server.py # Standard JSON-RPC stdio/SSE server (20 native tools)
├── tests/ # Exhaustive 1,883-test official verification suite
├── cli.py # CLI runner (extra doctor, test, indicators, run, inspect, snap)
├── install.ps1 # 1-line PowerShell installer (Windows)
├── install.sh # 1-line bash installer (macOS)
├── pyproject.toml # PyPI package manifest (extra-desktop v0.3.3)
├── package.json # npm package manifest (@yantraos/extra-desktop v0.3.3)
├── server.json # Official MCP registry manifest
├── glama.json # Glama MCP ecosystem manifest
├── smithery.yaml # Smithery 1-click deployment manifest
├── requirements.txt # Enterprise-audited dependency manifest
├── STARTER_PROMPT.md # Master AI system prompt & setup directive
├── docs/ # Full System Architecture & Specifications
└── RELEASE_NOTES.md # Comprehensive version release notes
```
---
## 🛡️ Enterprise Trust & Safety
* **Zero Unverified Binary Blobs:** No mysterious compiled `.dll` or `.pyd` files from solo maintainers.
* **Microsoft & Anthropic Standards:** Built exclusively on Microsoft system calls (`ctypes`), Anthropic's official `mcp` SDK, and PSF packages.
* **Fail-Safe Protection:** Includes a screen-corner emergency escape at `(0, 0)` and a global panic hotkey (`Ctrl+Alt+Shift+Q`).
---
## 🌌 Part of the yantraOS Sovereign Ecosystem
Extra is the Windows bridge for **[yantraOS](https://yantraos.com)**, the sovereign Arch Linux operating system engineered for autonomous computing.
If you want bare-metal AI autonomy with zero operating system telemetry, sub-microsecond OS kernel scheduling, and native Wayland hardware control, explore **[yantraos.com](https://yantraos.com)**.
---
## 🤝 Contributing
We welcome contributions from kernel hackers, automation researchers, and AI developers!
Please check out our [Contributing Guide](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md).
For vulnerability reporting, review our [Security Policy](SECURITY.md).
---
## 📜 License
Distributed under the MIT License. See [LICENSE](LICENSE) for details.
Copyright (c) 2026 Euryale Ferox Private Limited.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessResponsive