Peekaboo MCP: Lightning-fast macOS Screenshots & GUI Automation 🚀

🎉 NEW in v3: Complete GUI automation framework with AI Agent! Click, type, scroll, and automate any macOS application using natural language. Plus comprehensive menu bar extraction without clicking! See the GUI Automation section and AI Agent section for details.

Peekaboo is a powerful macOS utility for capturing screenshots, analyzing them with AI vision models, and now automating GUI interactions. It works both as a standalone CLI tool (recommended) and as an MCP server for AI assistants like Claude Desktop and Cursor.

🎯 Choose Your Path

🖥️ CLI Tool (Recommended for Most Users)

Perfect for:

• Command-line workflows and automation

• Shell scripts and CI/CD pipelines

• Quick screenshots and AI analysis

• System administration tasks

👻 MCP Server (For AI Assistants)

Perfect for:

• Claude Desktop integration

• Cursor IDE workflows

• AI agents that need visual context

• Interactive AI debugging sessions

What is Peekaboo?

Peekaboo bridges the gap between visual content on your screen and AI understanding. It provides:

• Lightning-fast screenshots of screens, applications, or specific windows

• AI-powered image analysis using GPT-4.1 Vision, Claude, Grok, or local models (Ollama)

• Complete GUI automation (v3) - Click, type, scroll, and interact with any macOS app

• Natural language automation (v3) - AI agent that understands tasks like "Open TextEdit and write a poem"

• Smart UI element detection - Automatically identifies buttons, text fields, links, and more with precise coordinate mapping

• Menu bar extraction (v3) - Discover all menus and keyboard shortcuts without clicking or opening menus

• Automatic session resolution - Commands intelligently use the most recent session (no manual tracking!)

• Window and application management with smart fuzzy matching

• Multi-screen support - List which screen windows are on and move them between displays

• Privacy-first operation with local AI options via Ollama

• Non-intrusive capture without changing window focus

• Automation scripting - Chain commands together for complex workflows

🏗️ Architecture

Peekaboo now composes three focused SwiftPM targets plus a thin umbrella:

• PeekabooAutomation – Screen capture, permissions, accessibility services, menu/window helpers, and every strongly typed model shared across apps.

• PeekabooAgentRuntime – Tool registry, MCP server tooling, streaming/agent glue. Nothing here touches AppKit directly; it consumes the automation protocols.

• PeekabooVisualizer – Event serialization and UI feedback for the macOS app + CLI visual overlays.

• PeekabooCore – _exported shim that re-exports the three modules so downstream targets can continue to import PeekabooCore while opting into narrower modules when desired.

• CLI – Command-line interface that injects a PeekabooServices() instance into commander commands and agents.

• Mac App – Native macOS GUI that keeps a long-lived @State private var services = PeekabooServices() and passes it through SwiftUI scenes.

• MCP Server – Model Context Protocol server for AI assistants (Claude Desktop, Cursor, etc.) built entirely on PeekabooAgentRuntime.

• Commander (in-repo) – Lightweight Swift 6 parsing helpers used by the CLI runtime (swift-tools-version 6.0, no Swift 6.2+ features).

The CLI command structs remain @MainActor so they run on the main thread, but the static commandDescription can just be a normal static let constant—no nonisolated(unsafe) or extra @MainActor wrappers are necessary when describing the command metadata.

All components share the same core services, ensuring consistent behavior and optimal performance. See Service API Reference for detailed documentation.

Embedding Peekaboo services

The legacy singleton is gone—if you construct PeekabooServices() yourself (e.g., tests, daemon, or a new host), call services.installAgentRuntimeDefaults() once after initialization so MCP tools, the ToolRegistry, and PeekabooAgentService share that instance:

Skipping the install step will cause MCP/ToolRegistry APIs to fatal with “default factory not configured” because there’s no hidden global anymore.

Git Submodules

Peekaboo vendors three shared dependencies as top-level git submodules:

Clone with git clone --recursive or run git submodule update --init --recursive after pulling to ensure all three are present.

Submodule Details (updated)

• AXorcist is our accessibility engine: it wraps the macOS AX APIs with type-safe Swift helpers so every command (and agent) can discover UI elements, grant permissions, and drive buttons/text fields without reinventing accessibility plumbing. Because the underlying APIs are macOS-only we keep AXorcist’s CI and release targets scoped to macOS 14+.

• Commander is the shared parser/runtime that replaces Swift Argument Parser across Peekaboo. It provides property-wrapper metadata, a central router, standard CLI flags, and binders that hydrate existing command structs while keeping the runtime @MainActor-friendly. Commander’s CI spans macOS, Linux, Apple simulators (iOS/tvOS/watchOS/visionOS), and Android (via --swift-sdk android).

• Tachikoma is the AI provider SDK plus MCP adapters. It now owns all credential/OAuth handling via TKAuthManager and exposes the standalone tk-config CLI (add/login/status) so hosts can reuse the same auth + validation flows. Peekaboo’s CLI delegates provider auth to Tachikoma; set TachikomaConfiguration.profileDirectoryName to .peekaboo to share credentials.

🧩 Platform Support

🚀 Quick Start: CLI Tool

Installation

Command Catalog

Peekaboo ships many focused commands. Each entry below links to a short doc in docs/commands/ with complete flag tables, workflows, and troubleshooting notes.

Vision & Capture

• see – Capture annotated UI maps, produce session IDs, and optionally run inline analysis.

• image – Grab raw PNG/JPG screenshots (screens, windows, menu bar) and feed them into AI with --analyze.

Core Utilities

• list – Subcommands: apps, windows, screens, menubar, permissions.

• tools – Enumerate native + MCP tools; filter by server/source.

• config – Subcommands: init, show (live validation), add <provider> <secret> [--timeout], login <provider> (OpenAI/Codex, Anthropic Max OAuth), legacy set-credential, models.

• permissions – status (default) and grant helpers for Screen Recording, Accessibility, etc.

• learn – Emit the full agent guide/system prompt/Commander signature dump.

• run – Execute .peekaboo.json automation scripts (--output, --no-fail-fast).

• sleep – Millisecond delays between scripted steps.

• clean – Prune session caches via --all-sessions, --older-than, or --session.

Interaction

• click – Element IDs, fuzzy queries, or coordinates; built-in wait/focus helpers.

• type – Text + escape sequences, --clear, tab/return/escape/delete flags.

• press – Special key sequences with repeat counts.

• hotkey – Modifier combos such as cmd,shift,t (terminal-safe parsing).

• scroll – Directional scrolls with optional element targets.

• swipe – Smooth drags between IDs or coordinates (--duration, --steps).

• drag – Drag-and-drop, modifiers, Dock/Trash targets.

• move – Cursor placement (coords, element IDs, queries, or screen center).

Windows, Menus, Apps, Spaces

• window – Subcommands: close, minimize, maximize, move, resize, set-bounds, focus, list.

• menu – click, click-extra, list, list-all for app menus and menu extras.

• menubar – list/click status-bar items by name or index.

• app – launch, quit, relaunch, hide, unhide, switch, list.

• open – macOS-style open with Peekaboo focus/failure handling.

• dock – launch, right-click, hide, show, list for Dock entries.

• dialog – click, input, file, dismiss, list system dialogs.

• space – list, switch, move-window (Spaces/virtual desktops).

Agents & Integrations

• agent – Natural-language automation (--dry-run, --resume, --model, audio options, session caching).

• mcp – serve, list, add, remove, enable, disable, info, test, call, inspect (stub) for Model Context Protocol workflows.

Each doc contains exhaustive flag descriptions and examples; the README only covers intent and grouping. Use peekaboo <command> --help for inline summaries.

Debugging with Verbose Mode

All Peekaboo commands support the --verbose or -v flag for detailed logging:

Verbose logs are written to stderr with timestamps:

This is invaluable for:

• Debugging automation scripts

• Understanding why elements aren't found

• Performance optimization

• Learning Peekaboo's internals

Configuration

Peekaboo uses a unified configuration directory at ~/.peekaboo/ for better discoverability:

Managing API Keys Securely

Example Configuration

~/.peekaboo/config.json:

~/.peekaboo/credentials (auto-created with proper permissions):

Common Workflows

👻 MCP Server Setup

For AI assistants like Claude Desktop and Cursor, Peekaboo provides a Model Context Protocol (MCP) server.

For Claude Desktop

1. Open Claude Desktop Settings (from the menubar, not the in-app settings)

2. Navigate to Developer → Edit Config

3. Add the Peekaboo MCP server configuration:

4. Save and restart Claude Desktop

For Claude Code

Run the following command:

Alternatively, if you've already installed the server via Claude Desktop, you can import it:

Local Development

For local development, use the built MCP server directly:

For Cursor IDE

Add to your Cursor settings:

🔗 MCP Client Integration

Peekaboo v3 now functions as both an MCP server (exposing its tools) and an MCP client (consuming external tools). This enables powerful workflows that combine Peekaboo's native automation with tools from the broader MCP ecosystem.

Default Integration: Chrome DevTools MCP

Peekaboo ships with the Chrome DevTools MCP enabled by default, providing Chromium automation via the DevTools protocol:

Managing External MCP Servers

Configuration

External servers are configured in ~/.peekaboo/config.json. To disable Chrome DevTools MCP:

Available External Tools

All external tools are prefixed with their server name:

• chrome-devtools:navigate_page - Navigate to URL (Chrome DevTools MCP)

• chrome-devtools:click - Click elements on webpage (Chrome DevTools MCP)

• chrome-devtools:screenshot - Take webpage screenshot (Chrome DevTools MCP)

• github:create_issue - Create GitHub issues (GitHub server)

• files:read_file - Read files (Filesystem server)

The AI agent automatically uses the best combination of native and external tools for each task.

See docs/mcp-client.md for complete documentation.

MCP Tools Available

Core Tools

1. image - Capture screenshots (with optional AI analysis via question parameter)

2. list - List applications, windows, or check server status

3. analyze - Analyze existing images with AI vision models (MCP-only tool, use peekaboo image --analyze in CLI)

UI Automation Tools

4. see - Capture screen and identify UI elements

5. click - Click on UI elements or coordinates

6. type - Type text into UI elements (supports escape sequences)

7. press - Press individual keys (return, tab, escape, arrows, etc.)

8. scroll - Scroll content in any direction

9. hotkey - Press keyboard shortcuts

10. swipe - Perform swipe/drag gestures

11. move - Move mouse cursor to specific position or element

12. drag - Perform drag and drop operations

Application & Window Management

13. app - Launch, quit, focus, hide, and manage applications

14. window - Manipulate windows (close, minimize, maximize, move, resize, focus)

15. menu - Interact with application menus and system menu extras

16. dock - Launch apps from dock and manage dock items

17. dialog - Handle dialog windows (click buttons, input text)

18. space - Manage macOS Spaces (virtual desktops)

Utility Tools

19. run - Execute automation scripts from .peekaboo.json files

20. sleep - Pause execution for specified duration

21. clean - Clean up session cache and temporary files

22. permissions - Check system permissions (screen recording, accessibility)

23. agent - Execute complex automation tasks using AI

📟 CLI Command Reference

Full command descriptions now live in docs/cli-command-reference.md.

🚀 GUI Automation with Peekaboo v3

Peekaboo v3 introduces powerful GUI automation capabilities, transforming it from a screenshot tool into a complete UI automation framework for macOS. This enables AI assistants to interact with any application through natural language commands.

How It Works

The v3 automation system uses a see-then-interact workflow:

1. See - Capture the screen and identify UI elements

2. Interact - Click, type, scroll, or perform other actions

3. Verify - Capture again to confirm the action succeeded

🎯 The see Tool - UI Element Discovery

The see tool is the foundation of GUI automation. It captures a screenshot and identifies all interactive UI elements, assigning them unique Peekaboo IDs.

Discovering Available Screens

Before capturing specific screens, you can list all connected displays:

This command shows:

• Screen index: Use with see --screen-index or image --screen-index

• Display name: Built-in, External, or specific model names

• Resolution: Full screen resolution

• Position: Coordinates in the unified desktop space

• Scale factor: Retina display information

• Visible area: Usable area (excluding menu bar on primary screen)

Multi-Screen Capture

When capturing multiple screens, Peekaboo automatically saves each screen as a separate file:

• Primary screen: screenshot.png

• Additional screens: screenshot_screen1.png, screenshot_screen2.png, etc.

Display information (name, resolution) is shown for each captured screen:

Note: Annotation is automatically disabled for full screen captures due to performance constraints.

Element ID Format

• B1, B2... - Buttons

• T1, T2... - Text fields/areas

• L1, L2... - Links

• G1, G2... - Groups/containers

• I1, I2... - Images

• S1, S2... - Sliders

• C1, C2... - Checkboxes/toggles

• M1, M2... - Menu items

🖱️ The click Tool

Click on UI elements using various targeting methods:

⌨️ The type Tool

Type text with support for escape sequences:

Supported Escape Sequences

• \n - Newline/return

• \t - Tab

• \b - Backspace/delete

• \e - Escape

• \\ - Literal backslash

🔑 The press Tool

Press individual keys or key sequences:

Available Keys

• Navigation: up, down, left, right, home, end, pageup, pagedown

• Editing: delete (backspace), forward_delete, clear

• Control: return, enter, tab, escape, space

• Function: f1-f12

• Special: caps_lock, help

📜 The scroll Tool

Scroll content in any direction:

⌨️ The hotkey Tool

Press keyboard shortcuts:

👆 The swipe Tool

Perform swipe or drag gestures:

🖱️ The move Tool

Move the mouse cursor to specific positions or UI elements:

🎯 The drag Tool

Perform drag and drop operations between UI elements or coordinates:

🔐 The permissions Tool

Check macOS system permissions required for automation:

📝 The run Tool - Automation Scripts

Execute complex automation workflows from JSON script files:

Script Format (.peekaboo.json)

🎯 Automatic Window Focus Management

Peekaboo v3 includes intelligent window focus management that ensures your automation commands target the correct window, even across different macOS Spaces (virtual desktops).

How Focus Management Works

All interaction commands (click, type, scroll, menu, hotkey, drag) automatically:

1. Track window identity - Using stable window IDs that persist across interactions

2. Detect window location - Find which Space contains the target window

3. Switch Spaces if needed - Automatically switch to the window's Space

4. Focus the window - Ensure the window is frontmost before interaction

5. Verify focus - Confirm the window is ready before proceeding

Focus Options

All interaction commands support these focus-related flags:

Space Management Commands

Peekaboo provides dedicated commands for managing macOS Spaces:

Window Focus Command

For explicit window focus control:

Focus Behavior

By default, Peekaboo:

• Automatically focuses windows before any interaction

• Switches Spaces when the target window is on a different desktop

• Waits for focus to ensure the window is ready

• Retries if needed with exponential backoff

This ensures reliable automation across complex multi-window, multi-Space workflows without manual window management.

👻 AI Agent Automation

Peekaboo v3 introduces an AI-powered agent that can understand and execute complex automation tasks using natural language. The agent uses OpenAI's Chat Completions API with streaming support to break down your instructions into specific Peekaboo commands.

Setting Up the Agent

Two Ways to Use the Agent

1. Direct Natural Language (Default)

When you provide a text argument without a subcommand, Peekaboo automatically uses the agent:

2. Explicit Agent Command

Use the agent subcommand for more control and options:

How the Agent Works

1. Understands Your Intent - The AI agent analyzes your natural language request

2. Plans the Steps - Breaks down the task into specific actions

3. Executes Commands - Uses Peekaboo's automation tools to perform each step

4. Verifies Results - Takes screenshots to confirm actions succeeded

5. Handles Errors - Can retry failed actions or adjust approach

Real-World Examples

Agent Options

• --verbose - See the agent's reasoning and planning process

• --dry-run - Preview what the agent would do without executing

• --max-steps <n> - Limit the number of actions (default: 20)

• --model <model> - Choose OpenAI model (default: gpt-4-turbo)

• --json-output - Get structured JSON output

• --resume - Resume the latest unfinished agent session

• --resume <session-id> - Resume a specific session by ID

Agent Capabilities

The agent has access to all Peekaboo commands:

• Visual Understanding - Can see and understand what's on screen

• UI Interaction - Click buttons, fill forms, navigate menus

• Text Entry - Type text, use keyboard shortcuts

• Window Management - Open, close, minimize, arrange windows

• Application Control - Launch apps, switch between them

• File Operations - Save files, handle dialogs

• Complex Workflows - Chain multiple actions together

• Multiple AI Models - Supports OpenAI (GPT-4o, o3), Anthropic (Claude), and Grok (xAI)

Understanding Agent Execution

When you run an agent command, here's what happens behind the scenes:

Example Workflow

Debugging Agent Actions

Use --verbose to see exactly what the agent is doing:

Tips for Best Results

1. Be Specific - "Click the blue Submit button" works better than "submit"

2. One Task at a Time - Break complex workflows into smaller tasks

3. Verify State - The agent works best when it can see the current screen

4. Use Verbose Mode - Add --verbose to understand what the agent is doing

5. Set Reasonable Limits - Use --max-steps to prevent runaway automation

Resuming Agent Sessions

The agent supports resuming interrupted or incomplete sessions, maintaining full conversation context:

How Resume Works

1. Session Persistence - Each agent run creates a session with a unique ID

2. Thread Continuity - Uses OpenAI's thread persistence to maintain conversation history

3. Context Preservation - The AI remembers all previous interactions in the session

4. Smart Recovery - Can continue from any point, understanding what was already done

Resume Examples

⏸️ The sleep Tool

Pause execution between actions:

🪟 The window Tool

Comprehensive window manipulation for any application:

Window Actions

• close - Close the window (animated if has close button)

• minimize - Minimize to dock

• maximize - Maximize/zoom window

• move - Move to specific coordinates

• resize - Change window dimensions

• set-bounds - Set position and size in one operation

• focus - Bring window to front and focus

Targeting Options

• app - Target by application name (fuzzy matching supported)

• title - Target by window title (substring matching)

• index - Target by index (0-based, front to back order)

🖥️ Multi-Screen Support

Peekaboo v3 includes comprehensive multi-screen support for window management across multiple displays. When listing windows, Peekaboo shows which screen each window is on, and provides powerful options for moving windows between screens.

Screen Identification

When listing windows, each window shows its screen location:

Moving Windows Between Screens

Using Screen Index (0-based):

Using Screen Presets:

Combined Screen and Window Operations

You can combine screen movement with window positioning:

How It Works

• Unified Coordinate System: macOS uses a single coordinate space across all screens

• Smart Positioning: When moving windows between screens without explicit coordinates, windows maintain their relative position (e.g., a window at 25% from the left edge stays at 25% on the new screen)

• Screen Detection: Windows are assigned to screens based on their center point

• 0-Based Indexing: Screens are indexed starting from 0, matching macOS's internal ordering

Multi-Screen with AI Agent

The AI agent understands multi-screen commands:

📋 The menu Tool

Interact with application menu bars and system menu extras:

Menu Subcommands

• list - List all menus and their items (including keyboard shortcuts)

• list-all - List menus for the frontmost application

• click - Click a menu item (default if not specified)

• click-extra - Click system menu extras in the status bar

Key Features

• Pure Accessibility - Extracts menu structure without clicking or opening menus

• Full Hierarchy - Discovers all submenus and nested items

• Keyboard Shortcuts - Shows all available keyboard shortcuts

• Smart Discovery - AI agents can use list to discover available options

🚀 The app Tool

Control applications - launch, quit, focus, hide, and switch between apps:

🎯 The dock Tool

Interact with the macOS Dock:

💬 The dialog Tool

Handle system dialogs and alerts:

🧹 The clean Tool

Clean up session cache and temporary files:

Session Management

Peekaboo v3 uses sessions to maintain UI state across commands:

• Sessions are created automatically by the see tool

• Each session stores screenshot data and element mappings

• Sessions persist in ~/.peekaboo/session/<PID>/

• Element IDs remain consistent within a session

• Sessions are automatically cleaned up on process exit

Best Practices

1. Always start with - Capture the current UI state before interacting

2. Use element IDs when possible - More reliable than coordinate clicking

3. Add delays for animations - Use sleep after actions that trigger animations

4. Verify actions - Call see again to confirm actions succeeded

5. Handle errors gracefully - Check if elements exist before interacting

6. Clean up sessions - Use the clean tool periodically

Example Workflows

Login Automation

Web Search

Form Filling

Troubleshooting

1. Elements not found - Ensure the UI is visible and not obscured

2. Clicks not working - Try increasing wait_for timeout

3. Wrong element clicked - Use specific element IDs instead of queries

4. Session errors - Run clean tool to clear corrupted sessions

5. Permissions denied - Grant Accessibility permission in System Settings

Debugging with Logs

Peekaboo uses macOS's unified logging system. Use pblog to monitor logs:

Note: macOS redacts log values by default, showing <private>. See docs/pblog-guide.md and docs/logging-profiles/README.md for solutions.

🔧 Configuration

For full configuration + environment variable tables, see docs/configuration.md.

🎨 Setting Up Local AI with Ollama

Need fully local models or Ultrathink experimentation? Follow the dedicated playbook in docs/ollama.md for installation, recommended models, environment variables, and troubleshooting tips. At runtime you can point Peekaboo at your Ollama server with PEEKABOO_AI_PROVIDERS="ollama/llama3.3" peekaboo agent "…".

📋 Requirements

Peekaboo needs macOS 14.0+, Screen Recording permission, and (ideally) Accessibility access. Follow docs/permissions.md for step-by-step instructions plus performance tips.

🏗️ Building from Source

See docs/building.md for prerequisites, pnpm build commands, and release script pointers.

👻 Poltergeist

Use the watcher by following docs/poltergeist.md; it covers start/stop commands, tuning, and queue behavior.

🧪 Testing

Running Tests

Peekaboo uses Swift Testing framework (Swift 6.0+) for all test suites:

Testing the CLI

📚 Documentation

• API Documentation

• Contributing Guide

• Blog Post

🐛 Troubleshooting

Enable debug logging for more details:

For step-by-step debugging, use the verbose flag:

🛠️ Development

Poltergeist - Automatic CLI Builder

Peekaboo includes Poltergeist, an automatic build system that watches Swift source files and rebuilds the CLI in the background. This ensures your CLI binary is always up-to-date during development.

Key features:

• Watches all Swift source files automatically

• Smart wrapper script (./scripts/peekaboo-wait.sh) handles build coordination

• Exit code 42 indicates build failure - fix immediately

• See Poltergeist repository for full documentation

Building from Source

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository

2. Create a feature branch

3. Commit your changes

4. Push to the branch

5. Open a Pull Request

📝 License

MIT License - see LICENSE file for details.

👤 Author

Created by Peter Steinberger - @steipete

🙏 Acknowledgments

• Apple's ScreenCaptureKit for blazing-fast captures

• The MCP team for the Model Context Protocol

• The Swift and TypeScript communities

📊 Coverage

Coverage generated via xcrun llvm-cov report Apps/CLI/.build/arm64-apple-macosx/debug/peekabooPackageTests.xctest/Contents/MacOS/peekabooPackageTests -instr-profile Apps/CLI/.build/arm64-apple-macosx/debug/codecov/default.profdata. Because the CLI target depends on AXorcist, Commander, Tachikoma, and TauTUI, the figure reflects the aggregate workspace, even though automation-heavy suites remain disabled during headless runs.