# YaleAgent

Linux desktop automation agent — Wayland-native, zero-restrictions, AI-ready.

**YaleAgent** is a comprehensive desktop automation framework for Linux, built with verified 2026-era backends. It provides screen capture, input simulation, browser automation, OCR, AT-SPI accessibility, and an MCP server for AI integration.

## Architecture

```
┌─────────────────────────────────────────────────────┐
│  YaleAgent                                          │
│                                                     │
│  ┌──────────┐  ┌──────────┐  ┌────────────────────┐│
│  │ CLI      │  │ Dashboard│  │ MCP Server         ││
│  │ typer    │  │ FastAPI  │  │ JSON-RPC 2.0 stdio ││
│  └──────────┘  └──────────┘  └────────────────────┘│
│                                                     │
│  ┌─────────────────────────────────────────────────┐│
│  │ Command Catalog (68 tools across 11 categories) ││
│  └─────────────────────────────────────────────────┘│
│                                                     │
│  ┌─────────────────────────────────────────────────┐│
│  │ Services Layer                                  ││
│  │  desktop: Portal screenshot, ydotool input,     ││
│  │           wmctrl windows, pyatspi AT-SPI        ││
│  │  browser: Playwright Chrome lifecycle           ││
│  │  ocr:     Tesseract text recognition            ││
│  │  store:   JSON key-value persistence            ││
│  └─────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────┘
```

## Quick Start

```bash
# Clone
git clone https://github.com/yaleedhaque/yale-agent.git
cd yale-agent

# Setup (installs deps + verifies backends)
./scripts/setup.sh

# Activate venv
source .venv/bin/activate

# Run
yale-agent dashboard          # Web UI at http://127.0.0.1:8766
yale-agent mcp                # MCP stdio server (for AI agents)
yale-agent tools              # List all 47 registered tools
yale-agent run desktop.screenshot   # Capture screen
```

## Tool Categories

### Desktop (8 tools)
| Tool | Description |
|------|-------------|
| `desktop.screenshot` | Capture screen or window (XDG Portal / scrot) |
| `desktop.mouse_move` | Move mouse to absolute position (ydotool Wayland) |
| `desktop.mouse_click` | Click mouse button (left/middle/right) |
| `desktop.key_press` | Press key or chord (ctrl+c, Return) |
| `desktop.type` | Type text string |
| `desktop.list_windows` | List all open windows |
| `desktop.activate_window` | Focus a window by title |
| `desktop.atspi_tree` | Read accessibility tree (pyatspi) |

### Browser (20 tools)
| Tool | Description |
|------|-------------|
| `browser.start` | Launch Chrome instance |
| `browser.stop` | Close browser |
| `browser.goto` | Navigate to URL |
| `browser.screenshot` | Capture page screenshot |
| `browser.click` | Click element by CSS selector |
| `browser.type` | Type into element |
| `browser.press` | Press keyboard key |
| `browser.eval` | Execute JavaScript |
| `browser.get_text` | Get visible text |
| `browser.get_html` | Get HTML source |
| `browser.tabs` | List open tabs |
| `browser.new_tab` | Open new tab |
| `browser.switch_tab` | Switch to tab by index |
| `browser.scroll` | Scroll page |
| `browser.status` | Get browser status |
| `browser.a11y_snapshot` | Accessibility tree snapshot (depth/cap) |
| `browser.dom_elements` | Extract interactive DOM elements (clickables) |
| `browser.click_text` | Click a text element by its text content |
| `browser.wait_text` | Wait for text to appear before timeout |
| `browser.fill_placeholder` | Fill input matched by placeholder text |

### Vision (5 tools)
| Tool | Description |
|------|-------------|
| `vision.ocr` | OCR an image file |
| `vision.screen_ocr` | Screenshot + OCR in one call |
| `vision.find` | Find text on screen/image → clickable center (exact + fuzzy) |
| `vision.click_text` | Find text then click it (ydotool) |
| `vision.ocr_region` | OCR a screen region (OmniParser-style GUI detection) |

### Media (5 tools)
| Tool | Description |
|------|-------------|
| `media.play_pause` | Toggle play/pause (playerctl) |
| `media.next` | Next track |
| `media.previous` | Previous track |
| `media.seek` | Seek ±seconds in current track |
| `media.status` | Currently playing tracks across players |

### Audio (3 tools)
| Tool | Description |
|------|-------------|
| `audio.volume_get` | Current volume + mute state (wpctl) |
| `audio.volume_set` | Set volume (0–1 or ±delta) |
| `audio.mute` | Toggle mute, reports real post-state |

### Clipboard (2 tools)
| Tool | Description |
|------|-------------|
| `clipboard.get` | Read clipboard (wl-paste) |
| `clipboard.set` | Write clipboard (wl-copy) |

### Notify (2 tools)
| Tool | Description |
|------|-------------|
| `notify.send` | Desktop notification (notify-send) |
| `notify.test` | Test notification with default body |

### System (6 tools)
| Tool | Description |
|------|-------------|
| `sys.exec` | Execute shell command |
| `sys.ps` | List running processes |
| `sys.info` | Get system information |
| `sys.disk` | Disk usage for path |
| `sys.env` | Get environment variables |
| `sys.launch` | Launch a desktop app (gtk-launch / gio) |

### Filesystem (9 tools)
| Tool | Description |
|------|-------------|
| `fs.read` | Read file contents |
| `fs.write` | Write/create file |
| `fs.append` | Append to file |
| `fs.list` | List directory |
| `fs.search` | Search files by pattern |
| `fs.copy` | Copy file/directory |
| `fs.delete` | Delete file/directory |
| `fs.info` | Get file metadata |
| `fs.hash` | Hash a file (sha256/md5) |

### Memory (5 tools)
| Tool | Description |
|------|-------------|
| `memory.get` | Get value from store |
| `memory.set` | Set value |
| `memory.delete` | Delete key |
| `memory.search` | Search by query |
| `memory.list` | List keys in namespace |

### Network (3 tools)
| Tool | Description |
|------|-------------|
| `net.http` | Make HTTP request |
| `net.ping` | Ping a host |
| `net.dns` | DNS lookup |

## Backend Stack (Verified on this machine)

| Capability | Backend | Notes |
|------------|---------|-------|
| Screenshot | XDG Desktop Portal (`org.freedesktop.portal.Screenshot`) | Wayland-native, no extension |
| Screenshot (fallback) | scrot | X11 fallback |
| Mouse input | ydotool 0.1.8 | Direct `/dev/uinput`, user in `input` group |
| Keyboard input | ydotool | symbolic chords (`ctrl+c`, `Alt+F4`, `Return`) |
| Clipboard | wl-copy / wl-paste | Wayland-native (xclip fallback) |
| Notifications | notify-send | DBus notifications |
| Media keys | playerctl | Play/pause/next/prev/seek/status |
| Volume | wpctl (WirePlumber) | `@DEFAULT_AUDIO_SINK@`, pactl fallback |
| Window mgmt | GNOME Shell Introspect → wmctrl | X11 wmctrl is the fallback |
| App discovery | pyatspi (AT-SPI2) | `desktop.childCount`, `desktop[i]` |
| OCR | Tesseract 5.3 | exact + word-aligned fuzzy matching |
| Browser | Playwright + Chrome 152 | `/usr/bin/google-chrome-stable` |

## MCP Integration

YaleAgent is a first-class MCP server. Configure in your MCP client:

```json
{
  "yale-agent": {
    "command": "yale-agent",
    "args": ["mcp"],
    "cwd": "/path/to/yale-agent"
  }
}
```

All 68 tools are exposed via the MCP `tools/list` and `tools/call` methods.

## MCP / CLI

- CLI: `yale-agent run <tool> '{json}'` → JSON result on stdout.
- MCP: `yale-agent mcp` → JSON-RPC 2.0 over stdio (`protocolVersion` 2024-11-05).
- Dashboard: `yale-agent dashboard` → FastAPI at `http://127.0.0.1:8766`.

## Input Permissions (Wayland)

Mouse/keyboard injection uses `ydotool` writing to `/dev/uinput`. Add the user to
the `input` group so no root is needed:

```bash
sudo usermod -aG input $USER   # then log out/in
```

A udev rule (`/etc/udev/rules.d/99-yale-input.rules`) sets `/dev/uinput` to
`root:input 0660` so the group owns it.

## Dashboard

The web dashboard at `http://127.0.0.1:8766` provides:
- Tool catalog with one-click execution
- JSON editor for custom tool calls
- Execution log
- Health status

## Configuration

Default config at `config/default.json`. Override via environment variables:
- `YALECT_TIMEOUT` — command timeout (seconds)
- `YALECT_BROWSER_PATH` — Chrome binary path
- `YALECT_DASHBOARD_PORT` — dashboard port
- `YALECT_LOG_LEVEL` — logging level

## License

MIT