# An MCP based AI Agent for PC Control

**An MCP based AI Agent for PC Control** is an **Interactive System Control AI Agent** that operates a real Windows desktop using **human-like primitives only**: perception (screen capture, OCR), mouse, keyboard, gestures, and navigation. No shortcut APIs — plan → act → observe → explain.

---

## Why It Impresses

- **Human-like interaction**: Uses only MCP tools (capture, click, type, open Settings, etc.).
- **True agent loop**: Plan → Act → Observe → Adapt → Explain.
- **Screen-grounded reasoning**: OCR + UI detection to verify state.
- **Explainable actions**: Plain-language summaries of what changed and why.
- **Safe by design**: Confirmation for destructive actions; all steps logged.

---

## Architecture

```
User intent → Planning → Perception → UI navigation → Execution → Verification → Explanation
```

| Layer | Role |
|-------|------|
| **Frontend** | Chat UI (demo). Primary interface: **Cursor** chat with MCP. |
| **Agent** | Cursor AI + system prompt + MCP tools. Plan → Execute → Observe. |
| **Perception** | Screen capture (mss/PyAutoGUI), OCR (Tesseract/EasyOCR), UI detection. |
| **Execution** | Mouse, keyboard, gestures, navigation (Start, Settings, Search, File Explorer). |
| **Safety** | `log_action`, `ask_user_confirmation`, pause/cancel. |
| **Memory** | Last screen context, last action, user preferences. |

---

## Requirements

- **OS**: Windows 10/11
- **Python**: 3.10+
- **RAM**: 8 GB min (16 GB recommended)
- **Tesseract**: Install [Tesseract OCR](https://github.com/UB-Mannheim/tesseract/wiki) and add it to `PATH` (or set `TESSERACT_CMD`).
- **Cursor**: Latest Cursor IDE with MCP support.

---

## Setup

### 1. Clone and install

**Option A — venv (recommended for MCP)**

```bash
setup.bat
```

This creates `.venv`, activates it, and runs `pip install -r requirements.txt`. The project’s `.cursor/mcp.json` is set up to use `.venv\Scripts\python.exe`, so Cursor will use this environment for the MCP server.

**Option B — manual**

```bash
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
```

If you use a global Python instead of a venv, edit `.cursor/mcp.json` and set `"command": "python"` (or `"py"` on Windows) instead of the `.venv` path.

### 2. Tesseract (OCR)

- Download and install [Tesseract for Windows](https://github.com/UB-Mannheim/tesseract/wiki).
- Ensure `tesseract` is on `PATH`, or set `TESSERACT_CMD` to the executable path.

### 3. Configure MCP Server

**Project config (recommended for testing)**  
This repo includes `.cursor/mcp.json`. When you open the project folder in Cursor, the **pc-control-agent** MCP server is used automatically. Ensure your `python` (or `py`) has the project deps installed and is on `PATH`.

If `python` doesn’t work on Windows, edit `.cursor/mcp.json` and set `"command": "py"` instead of `"command": "python"`.

**Optional — uv**

```json
{
  "mcpServers": {
    "pc-control-agent": {
      "command": "uv",
      "args": ["run", "python", "-m", "mcp_server.server"]
    }
  }
}
```

**Optional — global config**  
Use `~/.cursor/mcp.json` and add the same `pc-control-agent` entry if you want it available outside this project. Use full paths in `args` if needed.

Reload Cursor (or restart) after changing MCP config.


## Environment Variables

This project requires environment variables that are **not included** in the repository for security reasons.

1. Copy the example file:

```bash
cp .env.example .env
```

On Windows Command Prompt:

```cmd
copy .env.example .env

```

Or PowerShell:

```powershell
Copy-Item .env.example .env
```

2. Open the `.env` file.

3. Replace the placeholder values with your own credentials.

Example:

```env
MISTRAL_API_KEY=hf_xxxxxxxxxxxxxxxxxxxxxxxxx
```


## Test

1. **Open** the project folder in Cursor (`File → Open Folder`).
2. **Install deps** in that environment (e.g. `pip install -r requirements.txt` or use a venv and ensure Cursor uses it).
3. **Reload Cursor** (e.g. `Ctrl+Shift+P` → “Developer: Reload Window”) so it picks up `.cursor/mcp.json`.
4. Open **Chat** (or **Composer**). Under **Tools**, confirm **pc-control-agent** appears and its tools are listed.
5. **Try a command**, e.g.:
   - *“Get screen resolution”* → agent uses `get_screen_resolution`.
   - *“Open Settings”* → agent uses `open_settings`.
   - *“Capture the screen”* → agent uses `capture_screen`.

If the server fails to start:

- **`ModuleNotFoundError: No module named 'mcp'`** — Dependencies aren’t installed for the Python Cursor uses. Run `setup.bat` (or `pip install -r requirements.txt` in your venv), and ensure `.cursor/mcp.json` uses that Python (e.g. `.venv\Scripts\python.exe`).
- **`spawn python ENOENT`** — Cursor can’t find `python`. Use the **full path** to your Python in `mcp.json` (e.g. `"${workspaceFolder}/.venv/Scripts/python.exe"` when using the project venv).
- Check the Cursor MCP logs (**Output** → “MCP”) and confirm `python -m mcp_server.server` runs from the project root in a terminal.

---

## Usage

1. **Enable MCP Server** in Cursor (MCP server `pc-control-agent` connected).
2. In Cursor chat, give a **high-level goal** (e.g. *Show battery usage*, *Open File Explorer at Downloads*).
3. The agent will:
   - **Plan** steps (e.g. Open Settings → Search “battery” → Navigate to usage).
   - **Execute** them one-by-one via MCP tools.
   - **Observe** with `extract_text_from_screen` / `capture_screen` / `detect_ui_elements`.
   - **Explain** what you’re seeing and what changed.

---

## MCP Tools (overview)

- **Perception**: `capture_screen`, `capture_active_window`, `extract_text_from_screen`, `detect_ui_elements`, `get_cursor_position`, `get_screen_resolution`
- **Mouse**: `move_cursor`, `click`, `double_click`, `right_click`, `click_and_hold`, `release_click`
- **Keyboard**: `type_text`, `press_key`, `press_key_combination`, `hold_key`, `release_key`
- **Gestures**: `swipe`, `swipe_from_to`, `two_finger_scroll`, `tap`, `long_press`, `pinch_in`, `pinch_out`
- **Navigation**: `open_start_menu`, `open_settings`, `open_search`, `alt_tab`, `switch_window`, `maximize_window`, `minimize_window`, `close_window`, `open_file_explorer`, `navigate_to_path`
- **Safety**: `ask_user_confirmation`, `log_action`, `pause_execution`, `resume_execution`, `cancel_execution`
- **Memory**: `store_last_screen_context`, `recall_last_screen_context`, `recall_last_action`, `store_user_preference`, `recall_user_preference`, `clear_context`

---

## Project layout

```
.
├── backend/           # Perception, execution, safety, memory
├── mcp_server/        # MCP server (FastMCP) exposing tools
├── frontend/          # Demo chat UI
├── agent/             # System prompt and agent docs
├── .cursor/rules/     # Cursor rules for agent behaviour
├── requirements.txt
└── README.md
```

---

## License

MIT.