app-screen-mcp

Why this exists

Most mobile automation breaks when scripts act without observing current screen state.

app-screen-mcp fixes that by combining:

• Accessibility structure from idb ui describe-all

• Real pixels from Simulator screenshots

• Deterministic interactions (tap, type_text, swipe, hardware buttons)

This gives agents a closed loop: observe, reason, act, verify.

What you get

• Simulator discovery and boot control

• App launch and termination by bundle ID

• Full normalized accessibility tree

• JPEG screenshots with size/quality controls

• Image-hash suppression to skip unchanged screenshots

• Semantic actions by text or accessibility ID

• Relative-coordinate taps for resolution-independent flows

• One-call screen summary (UI tree + optional screenshot)

Architecture

MCP Client / Agent
        |
        v
   app-screen-mcp
        |
        +--> xcrun simctl (devices, app lifecycle, screenshots)
        |
        +--> idb (UI tree, tap, swipe, type, hardware buttons)
        |
        v
   iOS Simulator

Prerequisites

• macOS with Xcode + iOS Simulator

• Node.js 18+

• idb tooling

Manual install for idb:

brew tap facebook/fb
brew install idb-companion
pip3 install fb-idb

Install

Option 1: one-step installer (recommended)

bash <(curl -fsSL https://raw.githubusercontent.com/xmuweili/app-screen-mcp/main/install.sh)

The script checks/installs:

• Xcode Command Line Tools

• Homebrew

• Node.js (18+)

• idb-companion

• fb-idb

• app-screen-mcp (global npm package)

Option 2: global npm install

npm install -g app-screen-mcp

Option 3: from source

git clone https://github.com/xmuweili/app-screen-mcp.git
cd app-screen-mcp
npm install
npm run build

Configure your MCP client

Claude Desktop

Config file: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "ios-simulator": {
      "command": "node",
      "args": ["/absolute/path/to/app-screen-mcp/dist/index.js"]
    }
  }
}

Cursor / VS Code MCP

Common config paths:

• .cursor/mcp.json

• .vscode/mcp.json

Use the key your client expects: mcpServers or mcp.servers.

{
  "mcpServers": {
    "ios-simulator": {
      "command": "node",
      "args": ["/absolute/path/to/app-screen-mcp/dist/index.js"]
    }
  }
}

Restart your MCP client after updating config.

Tool reference (15 tools)

Quick usage patterns

1) Observe without image (cheap)

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": false,
    "compact_tree": true
  }
}

2) Add image when visual confirmation is required

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": true,
    "max_dim": 720,
    "quality": 55
  }
}

3) Skip unchanged screenshots

{
  "name": "get_screen_summary",
  "arguments": {
    "include_image": true,
    "only_if_changed": true,
    "previous_image_hash": "<last_hash>"
  }
}

4) Stable interaction flow

1. get_screen_summary(include_image=false)
2. find_elements("Sign In")
3. tap_id("login_email") or tap_text("Email")
4. type_text("user@example.com")
5. tap_text("Sign In")
6. get_screen_summary(include_image=true, only_if_changed=true)

Permission prompts

Permission behavior is enforced by the MCP client, not this server.

Claude Code (CLI)

Allow this server in ~/.claude/settings.json:

{
  "permissions": {
    "allow": ["mcp__ios-simulator__*"]
  }
}

ios-simulator must match your MCP server name.

Codex CLI

Codex uses command-level approvals. Use “always allow” for recurring safe command prefixes when prompted.

Local development

npm install
npm run build
npm start

Main implementation:

• src/index.ts

Testing

The integration tests run against a real booted iOS Simulator.

npm test

Before running tests:

• Boot an iOS Simulator device.

• Build/install demo app under demo-app/ if not already installed.

Test helpers:

• tests/helpers/client.ts

• tests/helpers/ui.ts

• tests/helpers/vision.ts

Troubleshooting

• No iOS simulator is currently running

  • Boot a simulator first, or call boot_simulator.

• idb command failures

  • Verify idb-companion and fb-idb installation and PATH.

• Missing/weak element matches

  • Improve app accessibility labels and identifiers.

• GUI client cannot find node/module path

  • Use an absolute path in args instead of relying on shell PATH.

Contributing

Contributions are welcome.

For tool changes:

• Update tool schema in src/index.ts

• Implement tool handler in src/index.ts

• Add integration coverage in tests/demo-app.test.ts

• Update Tool Reference in this README

License

MIT