Skip to main content
Glama

AIsistent

MCP server for non-intrusive RDP automation. OCR, YOLO button detection, and click injection — zero footprint on the remote machine.

GUI mode (default)

RDP headless mode

Capture

macOS RDP window / MSS fullscreen

Direct RDP framebuffer (simple-rdp)

OCR

Apple Vision / EasyOCR

Apple Vision / EasyOCR

Detection

YOLO (CUDA / MPS / CPU)

YOLO (CUDA / MPS / CPU)

Click

pyautogui

RDP protocol input channel


Quick Start

# macOS (Apple Silicon)
pip install aistent[apple]

# Windows / Linux (CPU)
pip install aistent[cpu]

# Windows (NVIDIA CUDA)
pip install aistent[cuda]

# RDP headless (any OS)
pip install aistent[rdp]

# Everything
pip install aistent[all]

aisistent                          # GUI mode (default)
aisistent --mode rdp --host HOST   # RDP headless mode

Related MCP server: umbriel

Tools

Tool

Description

connect_rdp

Open a headless RDP connection (switches transport to RDP mode)

disconnect_rdp

Close the RDP connection and switch back to GUI mode

capture_rdp_screen

Capture screen via active transport (GUI or RDP)

run_apple_ocr

OCR: Apple Vision (Mac) or EasyOCR (CPU/CUDA)

detect_rdp_buttons

YOLOv8 button detection on CUDA / MPS / CPU

inject_rdp_click

Click injection at percentage-based coordinates

benchmark

Run performance benchmark on OCR + YOLO (returns JSON)

Configuration

Env var

Default

Description

AISISTENT_YOLO_WEIGHTS

models/weights/best.pt

Path to YOLO weights file

AISISTENT_TEMP_DIR

temp_captures/

Screenshot temp directory

RDP_HOST

RDP server hostname/IP (for headless mode)

RDP_USER

RDP username

RDP_PASS

RDP password

RDP_DOMAIN

RDP domain (optional)


Cross-Platform Hardware Detection

Hardware is auto-detected at import time in aisistent/platform.py:

Backend

Detection

dtype

Use Case

CUDA (NVIDIA)

torch.cuda.is_available()

float16

Windows/Linux with NVIDIA GPU

MPS (Apple)

torch.backends.mps.is_available()

float16

macOS Apple Silicon (M1–M4)

CPU

fallback

float32

Any OS, no GPU

Install GPU backends

# CUDA (NVIDIA)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# MPS (Apple) — included in default torch on macOS
pip install torch torchvision

Benchmark

Run a quick performance test from the command line:

aisistent-bench                                 # captures a real screenshot & benchmarks
aisistent-bench --synthetic                     # use synthetic image (no screen capture)
aisistent-bench --skip-ocr                      # YOLO only
aisistent-bench --image screenshot.png          # use your own image
aisistent-bench --device cpu                    # force CPU backend

Or via MCP tool call:

benchmark(image_base64: "")              # empty = real screenshot, or pass base64

Real-world performance (Apple MacBook M5 — MPS GPU)

Benchmark on a real 1920×1080 desktop screenshot with text, buttons, and UI elements:

Platform : macOS (Apple Silicon M5)
Device   : MPS
──────────────────────────────────────
Capture  :  0.22s
OCR      :  0.43s  —  102 texts detected
YOLO     :  0.76s  —   59 buttons detected
──────────────────────────────────────
Total    : ~1.4s

Step

Time

Throughput

Screen capture

~0.22s

Apple Vision OCR (Neural Engine)

~0.43s

~237 texts/sec

YOLOv8 inference (MPS float16)

~0.76s

~78 detections/sec

End-to-end

~1.4s

These numbers are from the same machine running both the MCP server and the benchmark — no overhead from network or RDP. On NVIDIA CUDA, YOLO inference is typically 0.3–0.5s (RTX 3060+).


MCP Client Integration

AIsistent implements the standard MCP (Model Context Protocol), so it works with any MCP client. Below are detailed setup instructions for each platform.


Hermes MCP

Hermes is an AI agent that uses MCP tools to interact with your computer.

1. Install AIsistent

# macOS (Apple Silicon — Apple Vision OCR + MPS GPU)
pip install aistent[apple]

# Windows/Linux CPU
pip install aistent[cpu]

# Windows with NVIDIA GPU
pip install aistent[cuda]

2. Locate Hermes config file

OS

Path

macOS

~/.config/hermes/config.json

Windows

%APPDATA%\hermes\config.json

Linux

~/.config/hermes/config.json

3. Add AIsistent to Hermes config

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

If AIsistent is not on your PATH, use the full path:

{
  "mcpServers": {
    "aisistent": {
      "command": "/path/to/venv/bin/aisistent",
      "type": "stdio"
    }
  }
}

4. Start Hermes

hermes

Hermes will auto-discover AIsistent's tools on startup. You should see:

👁️ AIsistent — capture_rdp_screen, run_apple_ocr, detect_rdp_buttons, inject_rdp_click, benchmark

Example: Hermes asks AIsistent to read the screen

> What's on my screen right now?

Hermes will:

  1. Call capture_rdp_screen → gets screenshot

  2. Call run_apple_ocr(image) → extracts all text

  3. Call detect_rdp_buttons(image) → finds buttons

  4. Returns a structured summary of what's on screen

Example: Hermes clicks a button via AIsistent

> Open Chrome and go to youtube.com

Hermes will:

  1. Call capture_rdp_screen → sees desktop

  2. Call detect_rdp_buttons(image) → finds Chrome icon coordinates

  3. Call inject_rdp_click(12.5, 8.3) → clicks Chrome

  4. Repeats capture → detect → click until done


OpenCode

OpenCode is an agentic CLI that also supports MCP tools.

1. Install AIsistent

pip install aistent[all]

2. Add to OpenCode config

Create or edit ~/.config/opencode/opencode.jsonc:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Or per-project, add to .opencode.jsonc in your project root:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

3. Verify it works

opencode

Then ask:

capture the screen and tell me what applications are open

OpenCode will call capture_rdp_screenrun_apple_ocr and return the result.


Claude Desktop

Claude Desktop supports MCP tools via its config file.

1. Locate Claude Desktop config

OS

Path

macOS

~/Library/Application Support/Claude/claude_desktop_config.json

Windows

%APPDATA%\Claude\claude_desktop_config.json

2. Add AIsistent

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

3. Restart Claude Desktop

Claude will show a hammer icon with AIsistent's available tools.


Cursor

Cursor IDE supports MCP tools.

1. Open Cursor settings

SettingsFeaturesMCP Servers

2. Add server

Name: AIsistent
Type: stdio
Command: aisistent

3. Use in chat

In Cursor's AI chat, type:

@aisistent capture the screen and detect buttons

Any MCP Client (generic stdio)

If your MCP client uses stdio transport, the configuration is always the same pattern:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

For HTTP/SSE transport instead of stdio:

# Start AIsistent as an SSE server on port 8100
python -c "from aisistent.server import mcp; mcp.run(transport='sse', port=8100)"

Then configure:

{
  "mcpServers": {
    "aisistent": {
      "url": "http://localhost:8100/sse",
      "type": "sse"
    }
  }
}

Headless RDP Transport

AIsistent supports two transport modes that can be switched at runtime:

Feature

GUI mode (default)

RDP headless mode

Local window needed

Yes (Microsoft Remote Desktop)

No

Capture method

screencapture / MSS

Direct RDP framebuffer via simple-rdp

Click method

pyautogui (local screen)

RDP input channel

macOS support

Full

Full (no XQuartz needed)

Linux support

MSS fullscreen

Full

Windows support

MSS fullscreen

Full

CLI mode

# GUI mode (default)
aisistent

# RDP headless with inline credentials
aisistent --mode rdp --host 192.168.1.100 --user admin --password secret

# RDP headless with environment variables
export RDP_HOST=192.168.1.100
export RDP_USER=admin
export RDP_PASS=secret
aisistent --mode rdp

MCP tools (switch at runtime)

connect_rdp(host="192.168.1.100", username="admin", password="secret")
capture_rdp_screen()     # → remote framebuffer, no local window
inject_rdp_click(50, 50) # → click sent via RDP protocol
disconnect_rdp()         # → back to GUI mode

Credentials precedence

Arguments > Environment variables (RDP_HOST, RDP_USER, RDP_PASS) > Config file

Install

pip install aistent[rdp]    # headless RDP only
pip install aistent[all]    # everything including RDP

winremote-mcp Integration

AIsistent works alongside winremote-mcp for comprehensive Windows remote management. Run both MCP servers:

aisistent &                              # AIsistent (stdio)
winremote-mcp --transport sse --port 8100 # winremote-mcp (SSE)

AIsistent handles the visual layer (OCR, detection, clicks) while winremote-mcp handles system operations (registry, services, processes, files, etc.).


Docs


Project Structure

AIsistent/
├── aisistent/
│   ├── __init__.py      # Version
│   ├── __main__.py      # Entry point (argparse: --mode gui|rdp)
│   ├── server.py        # MCP server + tools
│   ├── platform.py      # OS + device detection
│   ├── config.py        # Settings management
│   ├── capture.py       # Screen capture (delegates to transport)
│   ├── ocr.py           # OCR (Apple Vision / EasyOCR)
│   ├── detection.py     # YOLOv8 button detection
│   ├── action.py        # Click injection (delegates to transport)
│   ├── benchmark.py     # Performance benchmark
│   └── transport/       # Pluggable transport layer
│       ├── __init__.py  # get/set transport singleton
│       ├── base.py      # Abstract Transport class
│       ├── gui.py       # GUI transport (screencapture + pyautogui)
│       └── rdp.py       # RDP headless transport (simple-rdp)
├── docs/                # Documentation
├── pyproject.toml
└── README.md

License

MIT


🇪🇸 AIsistent

Servidor MCP para automatización RDP no intrusiva. OCR, detección de botones con YOLOv8 e inyección de clics — sin instalar nada en la máquina remota.

Modo GUI (default)

Modo RDP headless

Captura

Ventana RDP macOS / MSS pantalla completa

Framebuffer RDP directo (simple-rdp)

OCR

Apple Vision / EasyOCR

Apple Vision / EasyOCR

Detección

YOLO (CUDA / MPS / CPU)

YOLO (CUDA / MPS / CPU)

Click

pyautogui

Canal de input RDP

Inicio Rápido

# macOS (Apple Silicon)
pip install aistent[apple]

# Windows / Linux (CPU)
pip install aistent[cpu]

# Windows (NVIDIA CUDA)
pip install aistent[cuda]

# RDP headless
pip install aistent[rdp]

aisistent                          # modo GUI
aisistent --mode rdp --host HOST   # modo RDP headless

Benchmark

aisistent-bench                          # pantallazo real
aisistent-bench --synthetic              # imagen sintética
aisistent-bench --image captura.png      # imagen propia

Resultados reales (MacBook M5 — MPS)

Paso

Tiempo

Elementos

Captura

~0.22s

OCR (Apple Vision)

~0.43s

102 textos

YOLO (MPS float16)

~0.76s

59 botones

Total

~1.4s

Transporte RDP headless

aisistent --mode rdp --host 192.168.1.100 --user admin --password pass
# O vía tool MCP:
# connect_rdp(host="...", username="...", password="...")
# disconnect_rdp()

Integración con MCP Clients

Hermes MCP

Añade AIsistent como servidor MCP en ~/.config/hermes/config.json:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Luego inicia Hermes: hermes

OpenCode

Añade en ~/.config/opencode/opencode.jsonc:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Claude Desktop

Añade en ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Licencia

MIT

Available Tools

4 tools
benchmarkA

Run performance benchmark on OCR + YOLO.

Args: image_base64: Optional base64-encoded PNG. If empty, generates a synthetic image.

ParametersJSON Schema
NameRequiredDescriptionDefault
image_base64No

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses that omitting image_base64 generates a synthetic image, which is useful. However, it does not detail other behavioral aspects like resource consumption, side effects, or measurement details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: two sentences that front-load the purpose and include parameter details. No extraneous words; every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that an output schema exists, the description need not detail return values. It provides enough context for a benchmark tool, though additional details about what the benchmark measures (e.g., timing, accuracy) would improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description compensates by explaining that image_base64 is optional and that an empty value triggers synthetic image generation. This adds clear meaning beyond the schema's type and default.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb-resource combination ('Run performance benchmark on OCR + YOLO'), clearly distinguishing it from sibling tools like capture_rdp_screen, run_apple_ocr, and inject_rdp_click.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives. Usage is implied for performance testing of OCR and YOLO, but no exclusions or context are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

capture_rdp_screenA

Capture the RDP window (or full screen as fallback).

macOS: captures Microsoft Remote Desktop window via Quartz WindowID. Windows/Linux: captures primary monitor via MSS (pip install aistent[all]).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description covers OS-specific behavior and fallback but does not disclose return format, permissions, or side effects. It is sufficient for a simple capture tool but lacks depth.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the main purpose, and efficiently structures OS-specific details. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple with no parameters and an output schema present. The description covers core behavior, OS differences, and fallback, making it complete for its complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters exist, so schema coverage is 100%. The description adds no parameter info (unnecessary), earning a baseline 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it captures the RDP window with a fallback to full screen, specifying the exact action and resource. It distinguishes from siblings like inject_rdp_click and run_apple_ocr by focusing on screen capture.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for capturing RDP screen states but does not explicitly state when to use this tool over siblings or mention prerequisites. The context is adequate but not explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

inject_rdp_clickB

Inject a mouse click at percentage-based screen coordinates.

ParametersJSON Schema
NameRequiredDescriptionDefault
x_percentYes
y_percentYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, and the description lacks details about side effects, such as whether it's a left-click, double-click, or if there's any waiting period. Very minimal behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence with no unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity and lack of annotations, the description is too sparse—missing details like button type, click behavior, and coordinate origin, which an agent needs for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description clarifies that coordinates are percentage-based, which adds meaning beyond the parameter titles, but does not specify valid ranges or origin (e.g., top-left).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool injects a mouse click using percentage-based coordinates, which distinguishes it from siblings like capture_rdp_screen and run_apple_ocr.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives, nor any prerequisites or when-not-to-use conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_apple_ocrA

Run OCR on a base64-encoded image.

macOS: Apple Vision OCR on Neural Engine (~0.05s). Other: EasyOCR on CPU or CUDA GPU (pip install aistent[all]).

ParametersJSON Schema
NameRequiredDescriptionDefault
image_base64Yes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the burden. It discloses the OCR engine per platform and approximate speed, but omits potential limitations like image size, format, or language support. The output schema covers return values, but behavioral caveats are missing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences long, with the core purpose in the first sentence and platform-specific detail in the second. No extraneous words or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the output schema exists, the description does not need to return values. It covers purpose, parameter, and platform behavior. For a single-parameter tool, it is reasonably complete, though additional context on image constraints would elevate it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The only parameter is 'image_base64', and the schema description coverage is 0%. The description adds 'base64-encoded image', which clarifies the string format but adds little beyond the parameter name. No additional constraints (e.g., max size) are given.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the verb ('Run OCR') and the resource ('base64-encoded image'), making the tool's purpose clear. It also distinguishes from siblings like benchmark and capture_rdp_screen, which are unrelated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides platform-specific guidance: on macOS it uses Apple Vision (fast, Neural Engine), on other platforms it uses EasyOCR (with optional GPU). This helps the agent understand performance expectations and installation requirements, though it stops short of explicit when-to-use advice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A3.7/5.0
Disambiguation5/5

All four tools have clearly distinct purposes: benchmark tests performance, capture grabs a screen, OCR extracts text, and inject_click interacts via mouse. No overlap in functionality.

Naming Consistency3/5

Naming is mixed: 'benchmark' is a noun while others use verb_noun (capture_rdp_screen, run_apple_ocr, inject_rdp_click). Also, prefixes are inconsistent—some include 'rdp' while others do not.

Tool Count4/5

Four tools is a reasonable number for a server focused on RDP automation and OCR. It covers the basic workflow without being overwhelming, though a few more would be welcome.

Completeness3/5

The toolset covers the core capture-OCR-click pipeline and includes a benchmark for testing. However, common automation actions like keyboard input or scrolling are missing, leaving notable gaps.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    B
    maintenance
    MCP server that provides computer control capabilities including mouse movements, keyboard actions, screenshot capture with OCR, and window management through a unified API.
    161
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    MCP server for driving any Windows app through five layers including OCR, UI Automation, and direct OS operations. Enables AI agents to control Windows desktop and OS cursor-free, even on background/locked windows.
    151
    2
    MIT
  • A
    license
    B
    quality
    B
    maintenance
    MCP server for Windows computer use, providing low-token, low-latency automation through UIA/Win32, OCR, and vision. It enables batch actions, shortcuts, isolated desktop execution, and browser control via MCP tools.
    13
    2
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/LaAlquimia/AIsistent'

If you have feedback or need assistance with the MCP directory API, please join our Discord server