optics-framework
OfficialDrives Android devices and emulators (including Android TV) to run automated test cases, locating and interacting with UI elements through the device's accessibility tree, OCR, or image matching.
Uses Appium as a mobile driver backend, enabling AI agents to execute keywords against Android and iOS apps and read back device/screen state through MCP resources.
Provides a BLE driver that controls production devices as a Bluetooth HID mouse/keyboard when debugging and screenshots are blocked, using coordinate-based interaction paired with screenshot and vision strategies.
Integrates Google Vision as an OCR engine for the text-detection locator strategy, finding UI elements on screenshots when the element tree cannot be used.
Automates iOS applications and devices via the framework's mobile driver backends, running the same keyword-based test steps used for Android.
Enables OpenAI-powered natural-language test authoring and AI self-heal recovery, letting an LLM read the screen and re-enter the locator ladder when all locator strategies fail.
Targets Samsung Tizen TVs, allowing test keywords to be executed against Samsung smart TV apps through the mobile driver backend.
Drives web browsers through Selenium, letting agents execute test keywords against web applications and inspect the resulting page state.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Docs · Install · Quickstart · Playground · Keywords
Optics is open-source test automation for Android, iOS, web, smart TV and desktop apps. You write a test as data, in CSV or YAML. Each element can have more than one way to find it, and Optics tries them in order: XPath, text, OCR, image, then an optional AI recovery. When the app changes and one way breaks, the next way finds the element and the test continues.
Try it in your browser
These four pages on optics-framework.org need no install. Click an image to open the page.
Real Optics, in the browser
/playground loads the optics-framework wheel from PyPI on Pyodide. Edit the suite, break a locator, then run dry_run or execute on Birch, a demo shop app. The output and junit_output.xml come from Optics itself.
optics live, next to a phone
The homepage playground is a simulation of optics live. Type a step or tap the phone. Each step runs at once and shows how Optics found the element. /save writes the steps that passed into modules.csv and elements.csv. Under the phone, each step also shows as CSV, Python, Robot, an HTTP call and an MCP call.
A real run, frame by frame
/run replays a real run of the Birch suite against two releases of the app. Each frame is a file that the run wrote. Release 2.2 passes every step and fails the run, because the last screen shows "Session expired".
Explain a suite
/explain reads a suite row by row. It shows what each step does, how each value resolves and what optics dry_run reports for it.
Related MCP server: adb-mcp
Install
One line installs the optics command into its own environment under ~/.optics:
curl -fsSL https://optics-framework.org/install | sh # macOS, Linuxirm https://optics-framework.org/install.ps1 | iex # WindowsThen run optics quickstart. It asks what you want to test, installs the engine for it, makes a project, checks your setup and runs the first test. Each prompt has a default.
To use pip, you need Python 3.12 or newer. The core package has no drivers, OCR or AI engines. Add the extras that you need:
python3 -m venv .venv && source .venv/bin/activate
pip install "optics-framework[appium,easyocr]"Kind | Extras |
Drivers |
|
OCR |
|
AI |
|
MCP server |
|
Bundles |
|
optics setup --list shows the same engines, and optics setup --install appium easyocr installs them for the Optics version you have.
A driver extra installs only the Python client. A phone test also needs the Appium server, a device or emulator, and the platform tools.pytesseract needs the Tesseract program, and playwright needs its browsers (playwright install). See the install guide.
Do not use Conda witheasyocr. Its NumPy version conflicts with the one that Optics needs. Use venv.
Four ways to create a test
Write it. Elements, modules and test cases are plain CSV or YAML files. The same keywords are also a Python library and a Robot Framework library.
Record it. Run keywords one at a time in
optics livenext to your device. Each step that works is recorded, and/savewrites the recording into your suite as a module.Describe it. Press Ctrl+N in
optics liveand describe the flow. An AI model (Gemini, Claude, OpenAI or a local model) drives the device one step at a time, then keeps only the steps that mattered.Delegate it. Run
optics mcp installto register the MCP server with Claude Code, or addoptics mcpto any MCP client. Your agent looks, taps, types and checks through the tools. When it is done, it gets the steps it ran as a suite.
A test is data
A suite is three kinds of file. Optics finds each file by its CSV header or YAML keys, not by its folder.
my_test_project/
├── config.yaml
├── test_cases/test_cases.csv
├── modules/modules.csv
└── test_data/
├── elements.csv
├── error_definitions.csv # optional
└── input_templates/*.png # optional, for image matchingelements.csv gives a name to each element. Give a name more than one row and Optics tries each value in order.
Element_Name,Element_ID
email_field,//android.widget.EditText[@resource-id="email"]
email_field,Email
sign_in,Sign in
sign_in,sign_in_button.pngA value can be an XPath, text=…, css=…, plain text, an image file from input_templates/, or TEXT_ONLY:… to search with OCR only.
modules.csv holds the steps, one keyword for each row. ${name} reads a value from elements.csv.
module_name,module_step,param_1,param_2
Sign In,Enter Text,${email_field},ava@birch.shop
Sign In,Press Element,${sign_in}test_cases.csv sets the modules that each test case runs, in order. A test case named Suite Setup or Suite Teardown runs before or after all the others.
test_case,test_step
Suite Setup,Launch Birch
Sign in and buy,Sign In
Sign in and buy,Add To CartHow Optics finds an element
Each keyword that acts on an element tries these ways in order and stops at the first one that works:
# | Way | What it does |
1 | XPath | Queries the accessibility tree through the driver |
2 | Text | Finds the visible text, CSS or class in the page source |
3 | OCR | Reads the screenshot with Apple Vision, RapidOCR, EasyOCR, Tesseract, Google Vision or a remote OCR server |
4 | Image | Matches a reference picture of the element on the screenshot |
5 | AI self-heal (optional) | An LLM reads the screen and recovers with the same keywords |
The fast ways run first, so OCR and image matching cost time only when the tree cannot help. Two more fallbacks run next to this one: the values of one element name, and the drivers or element sources that you enable, in config.yaml order.
Run a suite
optics dry_run my_test_project # check keywords, parameters, elements and modules without a device
optics execute my_test_project # run the suiteEach run writes to my_test_project/execution_output/:
junit_output.xml, written once when the run ends, also when you stop the run.Screenshots of each step that finds an element, with a box around what Optics found.
Detected errors. Put an
error_definitions.csvin the project and Optics reads the last screen for crash dialogs,Session expiredand other messages. A match adds a failure to the JUnit report, with no assertion. See error detection.
optics execute exits with 1 when a test fails, so CI marks the build as failed. See Run in CI for GitHub Actions and GitLab.
One suite, many targets
Target | Driver |
Android, iOS |
|
Web |
|
Android TV, Samsung Tizen, LG webOS |
|
macOS, Linux and Windows apps (Windows is experimental) |
|
Production devices that block debugging and screenshots |
|
config.yaml sets the driver and how to find elements. Each section is a list in order of priority, and each entry has an enabled flag. Enable a second driver and it becomes a fallback.
driver_sources:
- appium:
enabled: true
url: "http://localhost:4723"
capabilities:
platformName: Android
automationName: UiAutomator2
deviceName: emulator-5554
elements_sources:
- appium_find_element: { enabled: true }
- appium_page_source: { enabled: true }
- appium_screenshot: { enabled: true }
text_detection:
- easyocr: { enabled: true }optics configure <folder> writes this file from a few questions. See the configuration reference for every key, the AI engines and self-heal.
Use the keywords from code
The same keywords work from each of these:
Surface | Start it with |
CLI runner |
|
Interactive session |
|
Python |
|
Robot Framework |
|
HTTP API |
|
MCP server |
|
from optics_framework import Optics
optics = Optics()
optics.setup(
driver_sources=[{"appium": {"enabled": True, "url": "http://localhost:4723"}}],
elements_sources=[{"appium_find_element": {"enabled": True}}],
)
optics.launch_app(app_identifier="com.example.app")
optics.enter_text("username_field", "testuser")
optics.press_element("submit_button")
optics.validate_element("welcome_message")
optics.quit(){ "mcpServers": { "optics": { "command": "optics", "args": ["mcp"] } } }The server gives the agent 11 tools: start_session, observe, tap, type_text, scroll, press_key, launch_app, batch, check, run_keyword and terminate_session. Each action replies with what changed on the screen. For a network client, run optics mcp --transport http --port 8090. See the MCP reference.
Keywords
In CSV and YAML, keywords are in Title Case (Press Element). In Python they are in snake case (press_element).
Kind | Keywords |
Actions | Press Element · Press By Percentage · Press By Coordinates · Detect And Press · Select Dropdown Option · Swipe · Swipe By Percentage · Swipe From Element · Swipe Until Element Appears · Scroll · Scroll From Element · Scroll Until Element Appears · Enter Text · Enter Text Direct · Enter Text Using Keyboard · Enter Number · Clear Element Text · Press Keycode · Go Back · Get Text · Sleep · Execute Script |
Checks | Assert Presence · Assert Visibility · Assert Text · Assert Equality · Validate Element · Validate Screen · Is Element · Get Interactive Elements · Get Screen Elements · Capture Screenshot · Capture Pagesource |
App | Launch App · Launch Other App · Start Appium Session · Initialise Setup · Get Driver Session Id · Close And Terminate App · Force Terminate App · Get App Version |
Flow | Execute Module · Run Loop · Condition · Read Data · Evaluate · Date Evaluate · Invoke API |
optics list prints every keyword, and optics list press_element prints the parameters of one. The keyword reference has an example for each.
CLI
optics quickstart Make a first project, step by step
optics init Make a project (--template for a sample)
optics configure Write or edit config.yaml
optics doctor Check engines, tools and project config (--check for CI)
optics setup Install engines (--list, --install)
optics list Print the keywords, or the parameters of one
optics dry_run Check a project without a device
optics execute Run a project (--include, --exclude)
optics live Run keywords one at a time on a device
optics generate Write pytest or Robot Framework code from a project
optics serve Start the HTTP API
optics mcp Start the MCP server (install registers it with Claude Code)
optics completion Install shell completionSee the CLI reference for every option.
Contribute
Bug reports and code are welcome. Larger tasks have the help wanted label. To report a bug or ask for a keyword, open an issue.
git clone git@github.com:mozarkai/optics-framework.git
cd optics-framework
pipx install poetry
poetry install --with dev,test
poetry run pytest
poetry run pre-commit run --all-filesThe contribution guide has the development setup, how to add a keyword or an engine, and the commit format. Report security issues as SECURITY.md tells you, not in a public issue.
License
Apache 2.0. See LICENSE.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
remote debug iOS/Android/Unity/Godot/Flutter/RN/Web on real-device.ui-tree/screenshots/taps,tests.
Drive real Android & iOS devices and web browsers from natural language for mobile + web QA. 290+ tools across device control, app management, automation sessions, browser automation, and flow recording / replay. Bearer-auth — get a token at robotactions.com → Profile → API Tokens.
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
Drive and test your React Native and Expo apps end-to-end on iOS and Android. Supports real devices, simulators, and emulators. Snapshots, taps, typing, swipes, screenshots, deep links, reloads, builds and runtime errors.
Related MCP Servers
AlicenseAqualityAmaintenanceEnables AI assistants to automate mobile app testing and development for iOS and Android through natural language interactions. Supports intelligent element identification, session management, automated test generation, and comprehensive device interactions including clicks, swipes, screenshots, and app management.3210,260 npm519Apache 2.0- AlicenseAqualityBmaintenanceDrives an Android emulator or a physical device over adb. The agent screenshots, reads the UI hierarchy with true device-pixel coordinates, taps and types, manages app lifecycle, reads logcat, and runs Gradle builds and tests.8124MIT
- AlicenseAqualityBmaintenanceEnables coding agents to verify work on live cloud devices by uploading builds, starting sessions, driving UI through UI-tree refs, reading logs, and taking screenshots on iOS, Android, Apple TV, Android TV and Roku.384MIT
- AlicenseNot gradedqualityCmaintenanceTurns a real cloud iPhone or Android device into an environment an agent drives through up to eighteen capability-gated tools for reading the screen, tapping, swiping, typing, pressing keys and launching apps. The device also records video and a timed trajectory of every action for later inspection.8Apache 2.0