Windows MCP Server

# AI Usage Guide - Windows MCP Server **Essential guide for AI assistants using Windows MCP Server** ## 🎯 Core Principle: ALWAYS Start with get_desktop_state ### ❌ WRONG Approach ``` User: "Click the Submit button" AI: [Guesses coordinates] mouse_click(x=500, y=300) Result: ❌ Misses the button, clicks wrong spot ``` ### ✅ CORRECT Approach ``` User: "Click the Submit button" AI: 1. get_desktop_state() → See all elements with labels 2. Find "Submit" button → Label 7 3. click_element(label=7) → ✓ Success! ``` --- ## 🚀 Workflow Pattern (USE THIS!) ### Pattern 1: UI Interaction (Most Common) ``` Step 1: get_desktop_state() ↓ Step 2: Analyze elements ↓ Step 3: click_element() or type_into_element() ↓ Step 4: Confirm success ``` **Example:** ``` User: "Fill the login form and submit" AI Actions: 1. get_desktop_state() → Found: Label 3=Email field, Label 4=Password field, Label 5=Login button 2. type_into_element(label=3, text="user@email.com") → ✓ Typed into Email field 3. type_into_element(label=4, text="password123") → ✓ Typed into Password field 4. click_element(label=5) → ✓ Clicked Login button Result: ✅ Form filled and submitted successfully ``` ### Pattern 2: UI Interaction with Vision ``` Step 1: get_desktop_state(use_vision=True) ↓ Step 2: See annotated screenshot with labels ↓ Step 3: Use labels to interact ``` **When to use vision:** - User says "show me", "I need to see" - Complex UI with many elements - Need to verify exact layout - Debugging interaction issues ### Pattern 3: System Operations ``` Step 1: Direct tool use (no state needed) ↓ Step 2: Execute operation ↓ Step 3: Confirm result ``` **Example:** ``` User: "Take a screenshot and save to Desktop" AI Actions: 1. screenshot(save_path="C:\\Users\\UserName\\Desktop\\screenshot.png") → ✓ Screenshot saved Result: ✅ Screenshot saved to Desktop ``` --- ## 📋 Tool Selection Guide ### Start Every Session With: ``` get_desktop_state() - Know what's on screen ``` ### For Clicking: ``` ✅ PREFER: click_element(label=N) ❌ AVOID: mouse_click(x, y) - coordinates change! ``` ### For Typing: ``` ✅ PREFER: type_into_element(label=N, text="...") ❌ AVOID: keyboard_type("...") - might type in wrong place! ``` ### For Screenshots: ``` ✅ USE: screenshot() - for quick captures ✅ USE: get_desktop_state(use_vision=True) - for UI analysis ``` ### For Window Management: ``` ✅ USE: list_windows() - see all windows ✅ USE: activate_window(title="...") - switch windows ``` ### For Applications: ``` ✅ USE: launch_application(path="notepad.exe") ✅ USE: kill_process(name="notepad.exe") ``` --- ## ⚡ Speed Optimization ### Cache Awareness ```python # First call - scans UI tree (1.2s) get_desktop_state() # Within 30 seconds - use cached labels click_element(label=5) # Fast! type_into_element(label=7, text="hello") # Fast! # After 30 seconds - you'll get a warning click_element(label=5) # Warning: Cache stale # → Best: Refresh state get_desktop_state() # Fresh scan ``` ### Batch Operations ```python # ✅ GOOD: Get state once, use multiple times get_desktop_state() click_element(label=3) type_into_element(label=5, text="data") click_element(label=7) # ❌ BAD: Getting state repeatedly get_desktop_state() click_element(label=3) get_desktop_state() # Unnecessary! type_into_element(label=5, text="data") get_desktop_state() # Unnecessary! ``` --- ## 🎯 Best Practices ### 1. Always Describe What You're Doing ``` ❌ "Clicking element 5" ✅ "Clicking element 5 (Submit button) at (450,300)" ``` ### 2. Confirm Success ``` After each operation, acknowledge: "✓ Typed email into element 3" "✓ Clicked Login button (element 7)" ``` ### 3. Handle Errors Gracefully ``` If error: "Error: Label 99 out of range (0-45)" Response: "I need to refresh the desktop state. Let me scan again." → get_desktop_state() ``` ### 4. Use Descriptive Actions ``` ❌ type_into_element(label=5, text="abc") ✅ type_into_element(label=5, text="user@email.com") "Typing email address into Email field (element 5)" ``` ### 5. Leverage clear_first and press_enter ``` # Replace existing text type_into_element(label=3, text="new text", clear_first=True) # Submit after typing type_into_element(label=5, text="search query", press_enter=True) ``` --- ## 🚫 Common Mistakes to Avoid ### Mistake 1: Using Coordinates Instead of Labels ``` ❌ WRONG: mouse_click(x=500, y=300) # What if window moves? ✅ CORRECT: get_desktop_state() click_element(label=7) # Always works! ``` ### Mistake 2: Not Refreshing Stale State ``` ❌ WRONG: get_desktop_state() # 10:00 AM [... 5 minutes later ...] click_element(label=5) # UI might have changed! ✅ CORRECT: get_desktop_state() # 10:00 AM [... 5 minutes later ...] get_desktop_state() # Refresh! click_element(label=5) # Accurate! ``` ### Mistake 3: Typing Without Focusing ``` ❌ WRONG: keyboard_type("hello") # Types into whatever is focused! ✅ CORRECT: type_into_element(label=5, text="hello") # Clicks to focus first! ``` ### Mistake 4: Ignoring Error Messages ``` ❌ WRONG: Error: "Invalid label 99" Response: Trying again with same label ✅ CORRECT: Error: "Invalid label 99" Response: "Let me refresh the desktop state to see current elements" → get_desktop_state() ``` ### Mistake 5: Not Using Vision When Needed ``` ❌ WRONG: User: "I don't know which button to click" AI: [Guesses blindly] ✅ CORRECT: User: "I don't know which button to click" AI: get_desktop_state(use_vision=True) "Here's the annotated screenshot showing all buttons with labels..." ``` --- ## 💡 Pro Tips ### Tip 1: Combine Operations ```python # Fill entire form in one flow get_desktop_state() type_into_element(label=3, text="john@email.com", clear_first=True) type_into_element(label=4, text="password123", clear_first=True) click_element(label=5) # Submit "✓ Form filled and submitted successfully" ``` ### Tip 2: Use Scrollable Elements ```python # Scroll before clicking if element is off-screen get_desktop_state(include_scrollable=True) # Check scroll position in output # Scroll if needed using mouse_scroll or keyboard ``` ### Tip 3: Handle Different Clicks ```python # Right-click for context menu click_element(label=5, button="right") # Double-click to open click_element(label=7, clicks=2) # Triple-click to select all click_element(label=3, clicks=3) ``` ### Tip 4: Type Special Characters ```python # All text is supported type_into_element(label=5, text="user@email.com") type_into_element(label=7, text="C:\\Users\\Documents\\file.txt") type_into_element(label=9, text="Hello! How are you?") ``` ### Tip 5: Leverage Keyboard Shortcuts ```python # For system-wide actions, use keyboard shortcuts keyboard_press(keys=["win", "r"]) # Open Run dialog keyboard_type(text="notepad") keyboard_press(keys=["enter"]) ``` --- ## 📊 Decision Tree ``` User Request ↓ Does it involve UI interaction? ├─ YES → get_desktop_state() first │ ├─ Need to see UI? → use_vision=True │ ├─ Need to click? → click_element(label=N) │ ├─ Need to type? → type_into_element(label=N, text="...") │ └─ Multiple actions? → Use same state for all │ └─ NO → Direct tool use ├─ Screenshot? → screenshot() ├─ System info? → get_system_info() ├─ Launch app? → launch_application() └─ System control? → shutdown/restart/lock ``` --- ## 🎓 Example Scenarios ### Scenario 1: Web Form ``` User: "Fill the contact form: Name=John, Email=john@test.com, Submit" AI Steps: 1. get_desktop_state() 2. Analyze: Found Name field (label 3), Email field (label 5), Submit button (label 8) 3. type_into_element(label=3, text="John") 4. type_into_element(label=5, text="john@test.com") 5. click_element(label=8) 6. Response: "✓ Contact form filled and submitted" ``` ### Scenario 2: Search Operation ``` User: "Search for 'Python tutorial' and press enter" AI Steps: 1. get_desktop_state() 2. Find search box (label 12) 3. type_into_element(label=12, text="Python tutorial", press_enter=True) 4. Response: "✓ Searched for 'Python tutorial'" ``` ### Scenario 3: Complex Navigation ``` User: "Open Chrome, go to GitHub, and screenshot the page" AI Steps: 1. launch_application(path="chrome.exe") 2. Wait 2 seconds for Chrome to open 3. get_desktop_state() 4. Find address bar (label 15) 5. type_into_element(label=15, text="github.com", press_enter=True) 6. Wait 3 seconds for page to load 7. screenshot(save_path="Desktop/github.png") 8. Response: "✓ Opened Chrome, navigated to GitHub, took screenshot" ``` ### Scenario 4: Debugging ``` User: "The button isn't working" AI Steps: 1. get_desktop_state(use_vision=True) 2. Response: "I can see the screen now. Here are all the buttons: - Label 5: 'Submit' button at (450, 300) - Label 7: 'Cancel' button at (550, 300) - Label 9: 'Reset' button at (350, 300) Which button are you trying to click?" ``` --- ## ⚙️ Configuration Tips ### For Users (Claude Desktop Setup) ```json { "mcpServers": { "windows-control": { "command": "python", "args": ["-m", "windows_mcp.server"] } } } ``` ### For AI: Know Your Limitations ``` ✅ CAN DO: - Click any visible UI element - Type into any text field - Take screenshots - Launch applications - Control system (shutdown, restart, etc.) ❌ CANNOT DO: - Interact with elements that are off-screen (scroll first!) - Click on elements that don't exist (check with get_desktop_state) - Read minds (always scan state first!) - Work without proper setup (user must configure MCP server) ``` --- ## 🎯 Summary: The Golden Rule ``` ┌─────────────────────────────────────┐ │ │ │ ALWAYS GET STATE BEFORE ACTION │ │ │ │ get_desktop_state() → FIRST │ │ click/type → SECOND │ │ │ └─────────────────────────────────────┘ ``` **Remember:** 1. 🔍 **See first** (get_desktop_state) 2. 🎯 **Act second** (click_element / type_into_element) 3. ✅ **Confirm third** (acknowledge success) 4. 🔄 **Refresh when needed** (>30 seconds or UI changed) **This makes you 10x faster and 100x more accurate!** 🚀 --- ## 📚 Quick Reference Card ``` ┌──────────────────────────────────────────────────┐ │ MOST USED PATTERNS │ ├──────────────────────────────────────────────────┤ │ 1. Click something: │ │ get_desktop_state() │ │ click_element(label=N) │ │ │ │ 2. Type something: │ │ get_desktop_state() │ │ type_into_element(label=N, text="...") │ │ │ │ 3. Fill form: │ │ get_desktop_state() │ │ type_into_element(label=3, text="field1") │ │ type_into_element(label=5, text="field2") │ │ click_element(label=7) │ │ │ │ 4. See UI: │ │ get_desktop_state(use_vision=True) │ │ │ │ 5. Screenshot: │ │ screenshot(save_path="path/to/file.png") │ └──────────────────────────────────────────────────┘ ``` Use this guide every time you interact with Windows! 🎯