Peekaboo MCP

--- description: globs: alwaysApply: false --- ### Meta Note This file, `safari.mdc`, serves as a repository for detailed working notes, observations, and learnings acquired during the process of automating Safari interactions, particularly for the MCP Inspector UI. It's intended to capture the nuances of trial-and-error, debugging steps, and insights into what worked, what didn't, and why. This contrasts with `mcp-inspector.mdc`, which is designed to be the concise, polished, and operational ruleset for future automated runs once a specific automation flow (like connecting to the MCP Inspector) has been stabilized and proven reliable. `mcp-inspector.mdc` should contain the 'final' working scripts and minimal necessary commentary, while `safari.mdc` is the space for the extended antechamber of discovery. --- ### Key Learnings and Observations from Safari Automation (MCP Inspector) #### 1. Managing Safari Windows and Tabs for the Inspector * **Objective:** Reliably direct Safari to the MCP Inspector URL (`http://127.0.0.1:6274`) in a predictable way, preferably using a single, consistent browser window and tab to avoid disrupting the user's workspace or losing context. * **Initial Challenges & Evolution: * Simply using `make new document with properties {URL:"..."}` could lead to multiple windows/tabs if not managed. * Attempts to close all existing Inspector tabs first (`repeat with w in windows... close t...`) were functional but could be overly aggressive if the user had other work in Safari. * Identifying and reusing an *existing specific tab* for the Inspector requires careful targeting (e.g., `first tab whose URL starts with "..."`). If this tab was from a previous, unconfigured session, just switching to it wasn't enough; it needed to be reloaded/reset. * **Refined & Recommended Approach (as implemented in `mcp-inspector.mdc`): ```applescript tell application "Safari" activate delay 0.2 -- Allow Safari to become the frontmost application if (count of windows) is 0 then -- No Safari windows are open, so create a new one. make new document with properties {URL:"http://127.0.0.1:6274"} else -- Safari has windows open; use the frontmost one. tell front window set inspectorTab to missing value try -- Check if a tab for the Inspector is already open in this window. set inspectorTab to (first tab whose URL starts with "http://127.0.0.1:6274") end try if inspectorTab is not missing value then -- An Inspector tab exists: set its URL again (to refresh/reset) and make it active. set URL of inspectorTab to "http://127.0.0.1:6274" set current tab to inspectorTab else -- No specific Inspector tab found: set the URL of the *current active tab*. set URL of current tab to "http://127.0.0.1:6274" end if end tell end if delay 1 -- Pause to allow the page to begin loading. end tell ``` This logic aims to use the existing front window and either reuse/refresh an Inspector tab or repurpose the current active tab, falling back to creating a new window only if Safari isn't open. #### 2. Clicking Elements Programmatically (The "Connect" Button Saga) * **The Core Challenge:** Programmatically clicking the "Connect" button in the MCP Inspector UI to initiate the server connection. * **Strategies Explored & Lessons: * **CSS Selectors (`querySelector`):** * Simple selectors like `[data-testid='env-vars-button']` worked for some buttons but required escaping single quotes in AppleScript: `do JavaScript "document.querySelector('[data-testid=\\\'env-vars-button\\']').click();"`. * A complex `querySelector` for the "Connect" button (e.g., `'button[data-testid*=connect-button], button:not([disabled])... > span:contains(Connect)...'.click()`) ran without JS error but didn't reliably establish the connection, suggesting it might not have found the exact interactable element or the click wasn't registering correctly. * **XPath (`document.evaluate`):** * **Highly Specific XPaths:** An initial XPath based on the rule (`//button[contains(., 'Connect') and .//svg[.//polygon[@points='6 3 20 12 6 21 6 3']]]`) was very difficult to embed correctly in AppleScript due to nested single quotes requiring complex escaping (`\'`). This often led to AppleScript parsing errors (`-2741`). * **`character id 39` for AppleScript String Construction:** To combat escaping issues, building the JavaScript string in AppleScript using `set sQuote to character id 39` for internal single quotes was effective for getting the AppleScript parser to accept the command. Example: ```applescript set sQuote to character id 39 set jsConnectText to "Connect" set specificXPath to "//button[contains(., " & sQuote & jsConnectText & sQuote & ") and .//svg[.//polygon[@points=" & sQuote & "6 3 20 12 6 21 6 3" & sQuote & "]]]" set jsCommand to "document.evaluate(" & sQuote & specificXPath & sQuote & ", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.click();" ``` While this made the AppleScript runnable, this very specific XPath still didn't reliably trigger the connection. * **Successful XPath:** The breakthrough came with a slightly less specific but more robust XPath: `//button[.//text()='Connect']`. This finds a button that *contains* a text node exactly matching "Connect". * JavaScript: `document.evaluate("//button[.//text()='Connect']", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.click();` * AppleScript embedding (note `\"` for JS string quotes): ```applescript set jsCommand to "document.evaluate(\"//button[.//text()='Connect']\", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.click();" do JavaScript jsCommand in front document ``` This method proved successful in clicking the button and establishing the connection. * **`dispatchEvent(new MouseEvent('click', ...))`:** This was tried as an alternative to `.click()` but did not yield a different outcome for the "Connect" button in this specific scenario. #### 3. JavaScript Construction and Execution in AppleScript * **`do JavaScript "..."`:** This is the fundamental command. * **String Literals and Escaping:** * If the AppleScript command itself is enclosed in double quotes (`"..."`), then any literal double quotes *within the JavaScript code* must be escaped as `\\"`. * Single quotes (`'`) within the JavaScript code usually do not need escaping in this context. * Example: `do JavaScript "var el = document.getElementById(\"myId\"); el.value = 'Hello\';"` * **Long/Multiline JavaScript:** * Concatenating multiple AppleScript string literals using `&` (and optionally `¬` for line continuation) can build up a long JavaScript command. However, this can be fragile if not every part is perfectly quoted and escaped. Often, AppleScript parsing errors (`-2741`) occur before the JS is even attempted. * For complex JS, it's often more robust to ensure the entire JavaScript code is a single, well-formed string literal from AppleScript's perspective. If the JS itself is very complex, pre-constructing parts of it in AppleScript variables (especially strings that need careful quoting, like XPaths) can help. * **Returning Values:** The `do JavaScript` command returns the result of the last JavaScript statement executed. This can be invaluable for debugging, e.g., `return 'Found element';` or `return element !== null;`. #### 4. Asynchronicity and Delays * **Essential `delay` commands (Strategic vs. Tactical):** * **Strategic Delay (Crucial):** A critical lesson was the necessity of a significant delay (e.g., ~5 seconds) *after* an external process like the MCP Inspector is launched (e.g., via `npx` in iTerm) and *before* Safari automation attempts to interact with its web UI. This allows the external process and its web server to fully initialize. Without this, Safari automation might target a page that isn't ready or fully functional, leading to failures. * **Tactical Delays (Within Safari UI Automation - Often Avoidable):** Initially, small `delay` commands were used within Safari AppleScripts after actions like clicks or page loads (e.g., `delay 0.25`, `delay 1`). While these can sometimes help ensure the DOM is updated, the latest successful runs showed that if the backend/server (Inspector) is fully ready (due to the strategic delay), rapid Safari UI interactions (form filling, sequential clicks) can often be performed reliably *without* these internal micro-delays. Removing them can speed up the automation if the underlying application is responsive enough. * **Context is Key:** The need for tactical delays depends on how quickly the web application updates its DOM and responds to JavaScript events. For the MCP Inspector, once it's running, its UI seems to respond quickly enough to handle a sequence of JavaScript commands without interspersed AppleScript delays, provided the commands themselves are valid and target the correct elements. * **Checking for Results:** When verifying an action (e.g., checking if `document.body.innerText.includes('Connected')`), it's vital that this check happens *after* the action has had a chance to complete and the UI to reflect the change. If running without tactical delays, this check should still be performed after the relevant JavaScript action that's supposed to cause the change. #### 5. MCP Inspector Specifics * **URL Consistency:** The MCP Inspector URL (`http://127.0.0.1:6274`) was found to be consistent between runs, simplifying Safari targeting. * **Server Logs in the Inspector UI:** It was confirmed that after the `macos-automator-mcp` server connects via the MCP Inspector, its startup and operational logs (e.g., `[macos_automator_server] [INFO] Starting...`) are displayed directly within the MCP Inspector's web interface in Safari. This is the primary place to check for these server-specific logs, rather than the iTerm console running the `npx @modelcontextprotocol/inspector` command (which shows the Inspector's own proxy/connection logs). The Safari UI shows "Connected" status, and the server logs within the UI provide detailed confirmation of the server's state. #### 6. Automating iTerm via AppleScript and Advanced Timing Considerations * **Full iTerm Automation via AppleScript:** Due to persistent issues with iTerm-specific MCP tools (e.g., `mcp_iterm_send_control_character`, `mcp_iterm_write_to_terminal` consistently failing with "Tool not found" errors), a robust AppleScript workaround was developed and successfully implemented to manage the iTerm portion of the MCP Inspector setup. This script handles: * Activating iTerm. * Ensuring a window is available. * Sending a Control-C command to the current session using `System Events` (for reliability, targeting the iTerm process) to terminate any running commands. * Writing the `npx @modelcontextprotocol/inspector` command to the iTerm session to start the inspector. * The successful AppleScript structure is as follows (and now part of `mcp-inspector.mdc`): ```applescript tell application "iTerm" activate if (count of windows) is 0 then create window with default profile delay 0.5 # Brief delay for window creation end if end tell delay 0.2 # Ensure iTerm is frontmost tell application "System Events" # Note: 'iTerm' process name might need to be 'iTerm2' for iTerm3+. tell process "iTerm" keystroke "c" using control down end tell end tell delay 0.2 # Pause after Ctrl-C tell application "iTerm" tell current window tell current session write text "npx @modelcontextprotocol/inspector" end tell end tell end tell ``` * **iTerm Process Name in System Events:** When using `System Events` to control iTerm (e.g., for `keystroke`), the `tell process "iTerm"` command might need to be `tell process "iTerm2"` if using iTerm version 3 or later, as the application's registered process name can vary. * **Reinforcing the Strategic Delay:** The success of running Safari UI automation steps *without* internal (tactical) delays is highly dependent on the *strategic* delay implemented *after* initiating the MCP Inspector in iTerm and *before* beginning any Safari interaction. A delay of approximately 5 seconds was found to be effective, allowing `npx` and the Inspector server to fully initialize. Attempting Safari automation too soon, especially without tactical delays, will likely result in failures as the web UI won't be ready or responsive. #### 7. Interacting with Shadow DOM (Advanced) * **Identifying Shadow DOM:** Some web UIs, including potentially parts of the MCP Inspector (especially complex, self-contained components like the tool details and results panels), may use Shadow DOM to encapsulate their structure and styles. Standard `document.querySelector` or `document.evaluate` calls from the main document context will *not* pierce these shadow boundaries. * **Symptoms of Shadow DOM:** If `document.body.innerText` seems to miss details of an active UI component, or if standard selectors fail for visible elements that are clearly part of a specific component, Shadow DOM may be in use. * **Accessing Elements within Shadow DOM (Conceptual JavaScript Approach):** To interact with elements inside a shadow root, you first need a reference to the host element, then access its `shadowRoot` property, and then query within that root. ```javascript // 1. Find the host element (custom element tag name, e.g., 'tool-details-panel') const hostElement = document.querySelector('your-shadow-host-tag-name'); if (hostElement && hostElement.shadowRoot) { const shadowRoot = hostElement.shadowRoot; // 2. Query within the shadowRoot for target elements const targetElementInShadow = shadowRoot.querySelector('#some-element-inside-shadow'); if (targetElementInShadow) { // targetElementInShadow.click(); // return targetElementInShadow.textContent; } else { // return 'Element not found within shadowRoot'; } } else { // return 'Shadow host not found or no shadowRoot attached'; } ``` * **Recursive Deep Query Helper (Conceptual):** For nested shadow DOMs or when the exact host is unknown, a recursive or iterative deep query function can be useful. This function would traverse the DOM, checking each element for a `shadowRoot` and searching within it. ```javascript function $deep(selector, rootNode = document) { const stack = [rootNode]; while (stack.length) { const currentNode = stack.shift(); if (currentNode.nodeType === Node.ELEMENT_NODE && currentNode.matches(selector)) { return currentNode; } if (currentNode.shadowRoot) { stack.push(currentNode.shadowRoot); } // Check children only if it's an Element or DocumentFragment (like a shadowRoot) if (currentNode.nodeType === Node.ELEMENT_NODE || currentNode.nodeType === Node.DOCUMENT_FRAGMENT_NODE) { if (currentNode.children) { // Ensure children property exists stack.push(...currentNode.children); } } } return null; } // Usage: const someButton = $deep('button.some-class-in-shadow'); ``` * **Challenges with AppleScript `do JavaScript`:** * **Return Value Limitations:** Complex objects (like DOM elements) or very large strings (like extensive `outerHTML`) returned from `do JavaScript` can sometimes result in `missing value` or empty strings in AppleScript, making debugging difficult. * **Debugging:** Direct console logging from `do JavaScript` is not visible to the AppleScript environment, complicating troubleshooting of JavaScript execution within Safari. * **Reliability:** For highly dynamic UIs with extensive Shadow DOM, the AppleScript `do JavaScript` bridge may not always be reliable enough for complex, multi-step interactions, especially when precise timing or access to nuanced DOM states is required. Direct API/tool calls, if available, are often more robust for verification in such cases. * **Discovering Shadow Host Tag Names:** If the specific tag name of a shadow host is unknown, one might attempt to list all elements that have a `shadowRoot`: ```javascript // JavaScript to be executed via AppleScript to list shadow host tag names // (Note: Return value handling by AppleScript needs to be robust, e.g., JSON stringify) // let hosts = [...document.querySelectorAll('*')]\ // .filter(el => el.shadowRoot)\ // .map(el => el.tagName);\ // return JSON.stringify(hosts);\ ``` However, successful execution and return of this data via AppleScript `do JavaScript` can be unreliable, as experienced in attempts to automate the MCP Inspector. These notes capture the iterative process and key takeaways from the Safari automation for the MCP Inspector. The successful methods are now enshrined in `mcp-inspector.mdc`, while this document provides the background and context. --- ### Meta-Level Collaboration & Rule Evolution Notes * **Rule Refinement for Readability (User Feedback):** Based on user feedback, the main operational rule file (`mcp-inspector.mdc`) was refactored to move lengthy scripts (like the Safari tab setup AppleScript) into an Appendix section (e.g., `[Setup Safari Tab for Inspector]`). This keeps the main flow of the rule concise and readable for both humans and models, while still providing the full implementation details in a structured way. The `safari.mdc` file is designated for the more verbose, evolutionary notes and debugging narratives. * **Tool Usage Preferences (User Feedback):** User indicated a preference for using the `edit_file` tool for modifying rule files (like `.mdc` files) rather than `claude_code`. This allows the user to review the diff in their IDE before the change is effectively applied by the AI. This preference will be honored for future rule file modifications.