┌─────────────────────────────────────────────────┐
│                                                 │
│   1. SEE        screenshot / screen_ocr         │
│      ↓          "What's on the screen?"         │
│                                                 │
│   2. THINK      AI reasons about the content    │
│      ↓          "I need to click the Save btn"  │
│                                                 │
│   3. ACT        click_at / type_text / press_key│
│                 "Click at (425, 300)"           │
│                                                 │
│      ↻ repeat                                   │
└─────────────────────────────────────────────────┘

npx -y macos-control-mcp

{
  "mcpServers": {
    "macos-control": {
      "command": "npx",
      "args": ["-y", "macos-control-mcp"]
    }
  }
}

claude mcp add macos-control -- npx -y macos-control-mcp

{
  "servers": {
    "macos-control": {
      "command": "npx",
      "args": ["-y", "macos-control-mcp"]
    }
  }
}

{
  "mcpServers": {
    "macos-control": {
      "command": "npx",
      "args": ["-y", "macos-control-mcp"]
    }
  }
}

{
  "macos-control": {
    "command": "npx",
    "args": ["-y", "macos-control-mcp"]
  }
}

{
  "mcpServers": {
    "macos-control": {
      "command": "npx",
      "args": ["-y", "macos-control-mcp"]
    }
  }
}

You: "Go to example.com/signup and fill in my details"

Agent:
1. open_url("https://example.com/signup")
2. screenshot() → sees the form
3. screen_ocr() → finds "Email" field at (300, 250)
4. click_at(300, 250) → clicks the email field
5. type_text("user@example.com")
6. find_text_on_screen("Submit") → gets button coordinates
7. click_at(350, 500) → submits the form
8. screenshot() → confirms success

You: "Change the font size to 16 in TextEdit"

Agent:
1. launch_app("TextEdit")
2. screenshot() → sees the app
3. get_ui_elements("TextEdit") → finds menu items
4. press_key("t", ["command"]) → opens Fonts panel
5. screenshot() → sees the font panel
6. find_text_on_screen("Size") → locates the size field
7. click_at(x, y) → clicks size field
8. type_text("16")
9. press_key("return")

You: "Copy the tracking number from the shipping email in Mail"

Agent:
1. launch_app("Mail")
2. screenshot() → sees the inbox
3. find_text_on_screen("Your order has shipped") → locates the email
4. click_at(x, y) → opens the email
5. screenshot() → sees the email content
6. screen_ocr() → extracts all text including tracking number
7. set_clipboard("1Z999AA10123456784") → copies to clipboard

You: "What do I have open right now?"

Agent:
1. screenshot() → captures the full screen
2. Describes what it sees: apps, windows, content
3. list_running_apps() → ["Safari", "Slack", "VS Code", "Spotify"]
4. "You have Safari open on Twitter, Slack with 3 unread messages, and VS Code editing index.ts"

You: "The submit button on this page isn't working, can you check what's happening?"

Agent:
1. screenshot() → sees the page
2. find_text_on_screen("Submit") → locates button at (500, 600)
3. get_ui_elements("Safari") → inspects accessibility tree
4. "The button has aria-disabled=true. There's a required field 'Phone' that's empty."
5. find_text_on_screen("Phone") → locates the field
6. click_at(x, y) → clicks the field
7. type_text("+1 555-0123")
8. click_at(500, 600) → clicks Submit again
9. screenshot() → confirms it worked

You: "Pause the music and turn the volume down"

Agent:
1. launch_app("Spotify")
2. screenshot() → sees Spotify is playing
3. find_text_on_screen("Pause") → locates pause button
4. click_at(x, y) → pauses playback
5. find_text_on_screen("Volume") → locates volume slider area
6. click_at(x, y) → adjusts volume
7. screenshot() → confirms paused and volume lowered

You: "Create a new folder on my Desktop called 'Project Assets' and move it to Documents"

Agent:
1. launch_app("Finder")
2. press_key("d", ["command", "shift"]) → opens Desktop
3. screenshot() → sees Desktop in Finder
4. press_key("n", ["command", "shift"]) → creates new folder
5. type_text("Project Assets")
6. press_key("return")
7. screenshot() → confirms folder created

You: "Send 'build is green, ready to deploy' in the #engineering channel on Slack"

Agent:
1. launch_app("Slack")
2. screenshot() → sees Slack
3. press_key("k", ["command"]) → opens Quick Switcher
4. type_text("engineering")
5. press_key("return") → opens #engineering
6. screenshot() → confirms channel is open
7. click_at(x, y) → clicks message input
8. type_text("build is green, ready to deploy")
9. press_key("return") → sends message
10. screenshot() → confirms sent

You: "Look up the current price of AAPL on Google Finance and copy it"

Agent:
1. open_url("https://google.com/finance/quote/AAPL:NASDAQ")
2. screenshot() → sees the page loading
3. screen_ocr() → reads all text on the page
4. Finds the price: "$187.42"
5. set_clipboard("$187.42")
6. "Copied AAPL price $187.42 to your clipboard"

You: "Take what's in my clipboard, search for it in Safari, and screenshot the results"

Agent:
1. get_clipboard() → "best mechanical keyboards 2025"
2. launch_app("Safari")
3. press_key("l", ["command"]) → focuses address bar
4. type_text("best mechanical keyboards 2025")
5. press_key("return") → searches
6. screenshot() → captures the search results
7. "Here are the search results for 'best mechanical keyboards 2025'"

You: "Turn on Dark Mode"

Agent:
1. launch_app("System Settings")
2. screenshot() → sees System Settings
3. find_text_on_screen("Appearance") → locates the option
4. click_at(x, y) → opens Appearance settings
5. screenshot() → sees Light/Dark/Auto options
6. find_text_on_screen("Dark") → locates Dark mode option
7. click_at(x, y) → enables Dark Mode
8. screenshot() → confirms Dark Mode is on