# screen-mcp

An MCP (Model Context Protocol) server that gives AI agents **eyes** on an
Omarchy / Hyprland Wayland desktop. It captures screenshots via `grim` and
offers optional image analysis powered by the **Gemini API** — useful when the
calling model has no native vision encoder.

## Features

| Tool | What it does | Returns |
|---|---|---|
| `list_windows` | Enumerate Hyprland windows via `hyprctl` | Text listing with addresses, titles, workspaces, sizes |
| `screenshot_region` | Capture a pixel rectangle `(x, y, width, height)` | Inline PNG image (base64) |
| `screenshot_window` | Capture a window by title, class, address, or `"focused"` | Inline PNG image (base64) |
| `screenshot_fullscreen` | Capture the entire screen | Inline PNG image (base64) |
| `analyze_image` | Send an image (base64 or file path) to Gemini for reasoning | Text response |
| `screenshot_and_analyze` | Capture a region and analyze it in one call | Text (Gemini response) |

Screenshots use `grim` (wlroots screencopy protocol) — no X11 required.

## Prerequisites

- **Omarchy, Hyprland, or any wlroots-based Wayland compositor**
- `grim` — Wayland screenshot tool
- `slurp` — (optional) interactive region selection helper
- `hyprctl` — Hyprland window/query CLI
- `jq` — (optional) used by some helper scripts

Check with:

```bash
grim --help && hyprctl clients -j | head -c 20
```

### For image analysis (optional)

- A **Gemini API key** — get one at [https://aistudio.google.com](https://aistudio.google.com)
- Export it in your environment:

```bash
export GEMINI_API_KEY="your-api-key-here"
```

Without the key, the screenshot tools still work; only `analyze_image` and
`screenshot_and_analyze` will return an error.

## Installation

### Via Claude Desktop

Add this to your Claude Desktop `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "screen-mcp": {
      "command": "npx",
      "args": ["-y", "screen-mcp"],
      "env": {
        "GEMINI_API_KEY": "your-api-key-here"
      }
    }
  }
}
```

### Via npx

```bash
npx -y screen-mcp

# Or install globally:
npm install -g screen-mcp
```

## Usage examples

### Capture a screen region

```
screenshot_region(x=100, y=200, width=800, height=600, include_cursor=true)
```

### Capture a specific window

First list available windows to get an address or title:

```
list_windows()
```

Then capture by title (substring match), address, or `"focused"`:

```
screenshot_window(window="Spotify")
screenshot_window(window="focused")
screenshot_window(window="0x557ba79b4900")
```

### Analyze an image with Gemini

```
analyze_image(
  image_path="/tmp/my-screenshot.png",
  prompt="What applications are visible in this screenshot?",
  model="gemini-2.5-flash"
)
```

Or pass base64-encoded image data directly:

```
analyze_image(
  image="<base64-encoded-image>",
  prompt="Describe what you see in this image.",
  mime_type="image/png"
)
```

### Capture and analyze in one call

```
screenshot_and_analyze(
  x=0, y=0, width=1920, height=1080,
  prompt="Count the number of windows open and list their titles.",
  scale=0.5,
  model="gemini-2.5-flash"
)
```

## Development

```bash
# Install deps
npm install

# Build
npm run build

# Run
npm start

# Development (recompile on change)
npm run dev
```

### How it works

- **Capture**: `src/capture.ts` wraps `grim` (screenshots) and `hyprctl` (window
  enumeration). Window capture first tries `grim -T <stableId>` (foreign-toplevel
  handle), falling back to `grim -g "<x>,<y> <w>x<h>"` (geometry from hyprctl).
- **Analysis**: `src/gemini.ts` uses the official `@google/genai` SDK. Images are
  passed inline as base64 to the Gemini API's `interactions.create` endpoint.
- **Server**: `src/index.ts` wires everything together as a stdio-transported MCP
  server using `@modelcontextprotocol/server`.

## License

MIT