How do I use yale-agent?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@yale-agent Take a screenshot of the current window and OCR it to extract the text" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

yale-agent

by yaleedhaque

Overview Schema Related Servers Score Discussions

Python

Local

YaleAgent

Linux desktop automation agent — Wayland-native, zero-restrictions, AI-ready.

YaleAgent is a comprehensive desktop automation framework for Linux, built with verified 2026-era backends. It provides screen capture, input simulation, browser automation, OCR, AT-SPI accessibility, and an MCP server for AI integration.

Architecture

┌─────────────────────────────────────────────────────┐
│  YaleAgent                                          │
│                                                     │
│  ┌──────────┐  ┌──────────┐  ┌────────────────────┐│
│  │ CLI      │  │ Dashboard│  │ MCP Server         ││
│  │ typer    │  │ FastAPI  │  │ JSON-RPC 2.0 stdio ││
│  └──────────┘  └──────────┘  └────────────────────┘│
│                                                     │
│  ┌─────────────────────────────────────────────────┐│
│  │ Command Catalog (68 tools across 11 categories) ││
│  └─────────────────────────────────────────────────┘│
│                                                     │
│  ┌─────────────────────────────────────────────────┐│
│  │ Services Layer                                  ││
│  │  desktop: Portal screenshot, ydotool input,     ││
│  │           wmctrl windows, pyatspi AT-SPI        ││
│  │  browser: Playwright Chrome lifecycle           ││
│  │  ocr:     Tesseract text recognition            ││
│  │  store:   JSON key-value persistence            ││
│  └─────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────┘

Related MCP server: computer-use-linux

Quick Start

# Clone
git clone https://github.com/yaleedhaque/yale-agent.git
cd yale-agent

# Setup (installs deps + verifies backends)
./scripts/setup.sh

# Activate venv
source .venv/bin/activate

# Run
yale-agent dashboard          # Web UI at http://127.0.0.1:8766
yale-agent mcp                # MCP stdio server (for AI agents)
yale-agent tools              # List all 47 registered tools
yale-agent run desktop.screenshot   # Capture screen

Tool Categories

Desktop (8 tools)

Tool	Description
`desktop.screenshot`	Capture screen or window (XDG Portal / scrot)
`desktop.mouse_move`	Move mouse to absolute position (ydotool Wayland)
`desktop.mouse_click`	Click mouse button (left/middle/right)
`desktop.key_press`	Press key or chord (ctrl+c, Return)
`desktop.type`	Type text string
`desktop.list_windows`	List all open windows
`desktop.activate_window`	Focus a window by title
`desktop.atspi_tree`	Read accessibility tree (pyatspi)

Browser (20 tools)

Tool	Description
`browser.start`	Launch Chrome instance
`browser.stop`	Close browser
`browser.goto`	Navigate to URL
`browser.screenshot`	Capture page screenshot
`browser.click`	Click element by CSS selector
`browser.type`	Type into element
`browser.press`	Press keyboard key
`browser.eval`	Execute JavaScript
`browser.get_text`	Get visible text
`browser.get_html`	Get HTML source
`browser.tabs`	List open tabs
`browser.new_tab`	Open new tab
`browser.switch_tab`	Switch to tab by index
`browser.scroll`	Scroll page
`browser.status`	Get browser status
`browser.a11y_snapshot`	Accessibility tree snapshot (depth/cap)
`browser.dom_elements`	Extract interactive DOM elements (clickables)
`browser.click_text`	Click a text element by its text content
`browser.wait_text`	Wait for text to appear before timeout
`browser.fill_placeholder`	Fill input matched by placeholder text

Vision (5 tools)

Tool	Description
`vision.ocr`	OCR an image file
`vision.screen_ocr`	Screenshot + OCR in one call
`vision.find`	Find text on screen/image → clickable center (exact + fuzzy)
`vision.click_text`	Find text then click it (ydotool)
`vision.ocr_region`	OCR a screen region (OmniParser-style GUI detection)

Media (5 tools)

Tool	Description
`media.play_pause`	Toggle play/pause (playerctl)
`media.next`	Next track
`media.previous`	Previous track
`media.seek`	Seek ±seconds in current track
`media.status`	Currently playing tracks across players

Audio (3 tools)

Tool	Description
`audio.volume_get`	Current volume + mute state (wpctl)
`audio.volume_set`	Set volume (0–1 or ±delta)
`audio.mute`	Toggle mute, reports real post-state

Clipboard (2 tools)

Tool	Description
`clipboard.get`	Read clipboard (wl-paste)
`clipboard.set`	Write clipboard (wl-copy)

Notify (2 tools)

Tool	Description
`notify.send`	Desktop notification (notify-send)
`notify.test`	Test notification with default body

System (6 tools)

Tool	Description
`sys.exec`	Execute shell command
`sys.ps`	List running processes
`sys.info`	Get system information
`sys.disk`	Disk usage for path
`sys.env`	Get environment variables
`sys.launch`	Launch a desktop app (gtk-launch / gio)

Filesystem (9 tools)

Tool	Description
`fs.read`	Read file contents
`fs.write`	Write/create file
`fs.append`	Append to file
`fs.list`	List directory
`fs.search`	Search files by pattern
`fs.copy`	Copy file/directory
`fs.delete`	Delete file/directory
`fs.info`	Get file metadata
`fs.hash`	Hash a file (sha256/md5)

Memory (5 tools)

Tool	Description
`memory.get`	Get value from store
`memory.set`	Set value
`memory.delete`	Delete key
`memory.search`	Search by query
`memory.list`	List keys in namespace

Network (3 tools)

Tool	Description
`net.http`	Make HTTP request
`net.ping`	Ping a host
`net.dns`	DNS lookup

Backend Stack (Verified on this machine)

Capability	Backend	Notes
Screenshot	XDG Desktop Portal (`org.freedesktop.portal.Screenshot`)	Wayland-native, no extension
Screenshot (fallback)	scrot	X11 fallback
Mouse input	ydotool 0.1.8	Direct `/dev/uinput`, user in `input` group
Keyboard input	ydotool	symbolic chords (`ctrl+c`, `Alt+F4`, `Return`)
Clipboard	wl-copy / wl-paste	Wayland-native (xclip fallback)
Notifications	notify-send	DBus notifications
Media keys	playerctl	Play/pause/next/prev/seek/status
Volume	wpctl (WirePlumber)	`@DEFAULT_AUDIO_SINK@`, pactl fallback
Window mgmt	GNOME Shell Introspect → wmctrl	X11 wmctrl is the fallback
App discovery	pyatspi (AT-SPI2)	`desktop.childCount`, `desktop[i]`
OCR	Tesseract 5.3	exact + word-aligned fuzzy matching
Browser	Playwright + Chrome 152	`/usr/bin/google-chrome-stable`

MCP Integration

YaleAgent is a first-class MCP server. Configure in your MCP client:

{
  "yale-agent": {
    "command": "yale-agent",
    "args": ["mcp"],
    "cwd": "/path/to/yale-agent"
  }
}

All 68 tools are exposed via the MCP tools/list and tools/call methods.

MCP / CLI

CLI: yale-agent run <tool> '{json}' → JSON result on stdout.
MCP: yale-agent mcp → JSON-RPC 2.0 over stdio (protocolVersion 2024-11-05).
Dashboard: yale-agent dashboard → FastAPI at http://127.0.0.1:8766.

Input Permissions (Wayland)

Mouse/keyboard injection uses ydotool writing to /dev/uinput. Add the user to the input group so no root is needed:

sudo usermod -aG input $USER   # then log out/in

A udev rule (/etc/udev/rules.d/99-yale-input.rules) sets /dev/uinput to root:input 0660 so the group owns it.

Dashboard

The web dashboard at http://127.0.0.1:8766 provides:

Tool catalog with one-click execution
JSON editor for custom tool calls
Execution log
Health status

Configuration

Default config at config/default.json. Override via environment variables:

YALECT_TIMEOUT — command timeout (seconds)
YALECT_BROWSER_PATH — Chrome binary path
YALECT_DASHBOARD_PORT — dashboard port
YALECT_LOG_LEVEL — logging level

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

No tool schema history has been recorded yet.

This server cannot be installed

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Melaya
org.melaya
Melaya is a remote MCP server. It gives an assistant hands on your own Android phone and browser: it reads the screen through the accessibility tree, then taps, types and navigates inside the apps and sites you allow-list, with no per-app API. It also builds, schedules and runs agent pipelines across 6k+ connected tools. OAuth 2.1, nothing to install.
Agent Orchestration App Automation Browser Automation
1
ToolAPI B
org.toolapi
Free public MCP for AI agents — 193 tools, 44 workflows. No API key.
Agent Orchestration Autonomous Agents
19322
Frenchie A
io.github.Lab94
OCR, transcription, file extraction, and image generation for AI agents via MCP.
AI & Machine Learning Image & Video Processing Speech Processing
72
AiPayGen — 65+ AI Tools as an MCP Server B
io.github.Damien829
65+ AI tools as MCP: research, write, code, scrape, translate, RAG, agent memory, workflows
Autonomous Agents RAG Systems Research & Data
42

Related MCP Servers

desk-mcp
OS Automation App Automation Command Line
KpihX
A
license
Not graded
quality
D
maintenance
A desktop automation MCP server that enables AI agents to interact with Linux environments through screenshots, window inspection, and input simulation. It provides tools for mouse control, keyboard input, and screen capture using xdotool and XDG Desktop Portals.
Updated 5 months ago (2026-03-29 16:43 UTC)
MIT
computer-use-linux
OS Automation Autonomous Agents
agent-sh
A
license
Not graded
quality
A
maintenance
Controls a real Linux desktop from any MCP host. Reads accessibility trees, takes screenshots, and drives clicks, scrolls, and keystrokes across GNOME, KDE/KWin, Hyprland, i3, and COSMIC.
Updated a day ago (2026-09-05 20:12 UTC)
528
487
MIT
mcp-linux-desktop
OS Automation Shell Access
wizardofweb125-lab
A
license
Not graded
quality
D
maintenance
Enables full Linux desktop control including windows, mouse, keyboard, clipboard, audio, screenshots, OCR, accessibility, and system management through MCP-compatible AI agents.
Updated 3 months ago (2026-06-10 23:34 UTC)
1
MIT
dual-wield-mcp
OS Automation Autonomous Agents
SpiritualMachines
A
license
Not graded
quality
A
maintenance
MCP server for controlling Linux desktops over Wayland, enabling AI agents to perform mouse, keyboard, window, and screenshot operations on Fedora KDE Plasma.
Updated a month ago (2026-07-29 15:09 UTC)
AGPL 3.0

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/yaleedhaque/yale-agent'

If you have feedback or need assistance with the MCP directory API, please join our Discord server