mcp-appium-visual is an AI-powered mobile automation platform with Model Context Protocol (MCP) integration. It enables seamless control of Android and iOS devices through Appium, featuring intelligent visual element detection and recovery.
Features
- Integration with Appium for device control
- Visual element detection and recovery (AI-based)
- MCP support for advanced agent-driven testing workflows
- Supports Android and iOS platforms
- Designed for use with AI agents for intelligent automation
Prerequisites
- Node.js (v14 or higher)
- Java Development Kit (JDK)
- Android SDK (for Android testing)
- Xcode (for iOS testing, macOS only)
- Appium Server
- Android device or emulator / iOS device or simulator
Environment Setup
Before executing any commands, ensure your environment variables are properly set up:
- Make sure your
.bash_profile, .zshrc or other shell configuration file contains the necessary environment variables:
# Example environment variables in ~/.bash_profile
export JAVA_HOME=/path/to/your/java
export ANDROID_HOME=/path/to/your/android/sdk
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools
- Source your environment file before running MCP-Appium:
source ~/.bash_profile # For bash
# OR
source ~/.zshrc # For zsh
Note: The system will attempt to source your .bash_profile automatically when initializing the driver, but it's recommended to ensure proper environment setup manually before running tests in a new terminal session.
For iOS testing, proper Xcode command line tools configuration is essential:
- Install Xcode command line tools if not already installed:
- Verify the installation and check the current Xcode path:
- If needed, set the correct Xcode path (especially if you have multiple Xcode versions):
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
- Accept Xcode license agreements:
sudo xcodebuild -license accept
- For iOS real device testing, ensure your Apple Developer account is properly configured in Xcode:
- Open Xcode
- Go to Preferences > Accounts
- Add your Apple ID if not already added
- Download the necessary provisioning profiles
- Set up environment variables for iOS development:
# Add these to your ~/.bash_profile or ~/.zshrc
export DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer"
export PATH="$DEVELOPER_DIR/usr/bin:$PATH"
- Source your updated configuration:
source ~/.bash_profile # For bash
# OR
source ~/.zshrc # For zsh
Setup
- Install dependencies:
- Install and start Appium server:
npm install -g appium
appium
- Set up Android device/emulator:
- Enable Developer Options on your Android device
- Enable USB Debugging
- Connect device via USB or start an emulator
- Verify device is connected using
adb devices
- For iOS testing (macOS only):
- Ensure Xcode command line tools are installed:
xcode-select --install - Set up iOS simulator or connect a real device
- Trust the development computer on the iOS device if using a real device
Running Tests
- Build the project:
- Start the MCP server:
- In a new terminal, run the test:
Test Configuration
Android Configuration
The example test uses the Android Settings app as a demo. To test your own app:
- Edit
examples/appium-test.ts:- Update
deviceName to match your device - Set
app path to your APK file, or - Update
appPackage and appActivity for an installed app
- Common capabilities configuration:
const capabilities: AppiumCapabilities = {
platformName: "Android",
deviceName: "YOUR_DEVICE_NAME",
automationName: "UiAutomator2",
// For installing and testing an APK:
app: "./path/to/your/app.apk",
// OR for testing an installed app:
appPackage: "your.app.package",
appActivity: ".MainActivity",
noReset: true,
};
iOS Configuration
For iOS testing using the new Xcode command line support:
- Example configuration in
examples/xcode-appium-example.ts:
const capabilities: AppiumCapabilities = {
platformName: "iOS",
deviceName: "iPhone 13", // Your simulator or device name
automationName: "XCUITest",
udid: "DEVICE_UDID", // Get this from XcodeCommands.getIosSimulators()
// For installing and testing an app:
app: "./path/to/your/app.app",
// OR for testing an installed app:
bundleId: "com.your.app",
noReset: true,
};
Available Actions
The MCP server supports various Appium actions:
- Element Interactions:
- Find elements
- Tap/click elements with W3C Actions API (See "W3C Standard Gestures" section)
- Type text
- Scroll to element with W3C Actions API
- Long press
- App Management:
- Launch/close app
- Reset app
- Get current package/activity
- Device Controls:
- Screen orientation
- Keyboard handling
- Device lock/unlock
- Screenshots
- Battery info
- Advanced Features:
- Context switching (Native/WebView)
- File operations
- Notifications
- Custom gestures
- Xcode Command Line Tools (iOS only):
- Manage iOS simulators (boot, shutdown)
- Install/uninstall apps on simulators
- Launch/terminate apps
- Take screenshots
- Record videos
- Create/delete simulators
- Get device types and runtimes
W3C Standard Gestures
The MCP-Appium library now implements the W3C WebDriver Actions API for touch gestures, which is the modern standard for mobile automation.
W3C Actions for Tap Elements
The tapElement method now uses the W3C Actions API with intelligent fallbacks:
// The method will try in this order:
// 1. Standard WebdriverIO click()
// 2. W3C Actions API
// 3. Legacy TouchAction API (fallback for backward compatibility)
await appium.tapElement("//android.widget.Button[@text='OK']");
// or using the click alias
await appium.click("//android.widget.Button[@text='OK']");
The scrollToElement method now uses W3C Actions API:
// Uses W3C Actions API for more reliable scrolling
await appium.scrollToElement(
"//android.widget.TextView[@text='About phone']", // selector
"down", // direction: "up", "down", "left", "right"
"xpath", // strategy
10 // maxScrolls
);
Custom W3C Gestures
You can create your own custom W3C gestures using the executeMobileCommand method:
// Create custom W3C Actions API gesture
const w3cActions = {
actions: [
{
type: "pointer",
id: "finger1",
parameters: { pointerType: "touch" },
actions: [
// Move to start position
{ type: "pointerMove", duration: 0, x: startX, y: startY },
// Press down
{ type: "pointerDown", button: 0 },
// Move to end position over duration milliseconds
{
type: "pointerMove",
duration: duration,
origin: "viewport",
x: endX,
y: endY,
},
// Release
{ type: "pointerUp", button: 0 },
],
},
],
};
// Execute the W3C Actions using executeScript
await appium.executeMobileCommand("performActions", [w3cActions.actions]);
See examples/w3c-actions-swipe-demo.ts for more examples of W3C standard gesture implementations.
The new XcodeCommands class provides powerful tools for iOS testing:
import { XcodeCommands } from "../src/lib/xcode/xcodeCommands.js";
// Check if Xcode CLI tools are installed
const isInstalled = await XcodeCommands.isXcodeCliInstalled();
// Get available simulators
const simulators = await XcodeCommands.getIosSimulators();
// Boot a simulator
await XcodeCommands.bootSimulator("SIMULATOR_UDID");
// Install an app
await XcodeCommands.installApp("SIMULATOR_UDID", "/path/to/app.app");
// Launch an app
await XcodeCommands.launchApp("SIMULATOR_UDID", "com.example.app");
// Take a screenshot
await XcodeCommands.takeScreenshot("SIMULATOR_UDID", "/path/to/output.png");
// Shutdown a simulator
await XcodeCommands.shutdownSimulator("SIMULATOR_UDID");
Using the Click Function
The click() method provides a more intuitive alternative to tapElement():
// Using the click method
await appium.click("//android.widget.Button[@text='OK']");
// This is equivalent to:
await appium.tapElement("//android.widget.Button[@text='OK']");
Troubleshooting
- Device not found:
- Check
adb devices output - Verify USB debugging is enabled
- Try reconnecting the device
- App not installing:
- Verify APK path is correct
- Check device has enough storage
- Ensure app is signed for debug
- Elements not found:
- Use Appium Inspector to verify selectors
- Check if elements are visible on screen
- Try different locator strategies
- Connection issues:
- Verify Appium server is running
- Check port conflicts
- Ensure correct capabilities are set
- iOS Simulator issues:
- Verify Xcode command line tools are installed:
xcode-select -p - Check simulator UDID is correct using
xcrun simctl list devices - Close and restart simulator if it becomes unresponsive
Contributing
Feel free to submit issues and pull requests for additional features or bug fixes.
License
MIT