Provides comprehensive Roku automated testing by combining Roku WebDriver and ECP for UI state verification and device control, enabling acceptance criteria validation for Roku channels.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@RokuHarness MCP Server Verify that a label with text 'Welcome' is present on screen"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A Model Context Protocol (MCP) server for comprehensive Roku automated testing. RokuHarness combines Roku WebDriver and ECP (External Control Protocol) to provide both UI state verification and device control - enabling true acceptance criteria validation, not just remote control simulation.
Capability | ECP | WebDriver |
Press remote buttons | ✅ | ✅ |
Launch apps | ✅ | ✅ |
Query UI elements | ❌ | ✅ |
Verify text is on screen | ❌ | ✅ |
Check element attributes | ❌ | ✅ |
Get SceneGraph XML | ❌ | ✅ |
Take screenshots | ❌ | ✅ |
Validate acceptance criteria | ❌ | ✅ |
Bottom line: ECP can press buttons, WebDriver can verify what happens.
Related MCP server: Roku MCP Server
Your Tests (via any MCP client)
↓
RokuHarness MCP Server (this project)
↓ (WebDriver HTTP API)
Roku WebDriver Server (from Roku's repo)
↓ (ECP + Debug APIs)
Roku Device (your sideloaded channel)Key Point: The Roku WebDriver Server uses BOTH ECP (for control) and Roku's debug APIs (for UI verification). This MCP server provides a unified interface to both capabilities.
You need to download and run Roku's official WebDriver server:
# Clone Roku's automated testing repo
git clone https://github.com/rokudev/automated-channel-testing.git
cd automated-channel-testing
# Build the WebDriver server (requires Go)
cd src
go build
# Run the server
./RokuWebDriver # Linux/Mac
# or
RokuWebDriver.exe # WindowsThe server will start on http://localhost:9000 by default.
Download pre-built binaries: Check the automated-channel-testing/bin folder for pre-compiled executables.
WebDriver requires your channel to be sideloaded in developer mode:
Enable developer mode on your Roku: Settings → System → About → Press Home 5x, Up, Rewind 2x, Fast Forward 2x
Package your channel as a .zip file
Visit http://YOUR_ROKU_IP in a browser
Upload and install your channel
Important: WebDriver only works with:
Sideloaded developer channels (app ID: dev)
Channels packaged with your developer account on that specific device
SceneGraph-based channels (not legacy BrightScript)
npm install
npm run buildnpm install
npm run buildFor Claude Desktop:
Add to claude_desktop_config.json:
{
"mcpServers": {
"rokuharness": {
"command": "node",
"args": ["/absolute/path/to/rokuharness-mcp/build/index.js"]
}
}
}For Custom MCP Clients:
RokuHarness is MCP-agnostic and works with any MCP client. See "Using Without Claude Desktop" section below for integration examples.
In a separate terminal:
cd /path/to/automated-channel-testing/bin
./RokuWebDriver # or RokuWebDriver.exe on WindowsKeep this running while testing.
The MCP server will now be available in Claude.
Every test session starts by creating a WebDriver session:
Create a Roku WebDriver session:
- WebDriver URL: http://localhost:9000
- Roku IP: 192.168.1.100
- App: devThis connects to your sideloaded channel.
Before writing any verification queries, inspect the UI:
Get the UI source to see what elements are on screenYou'll get back SceneGraph XML like:
<Scene>
<Label id="titleLabel" text="Welcome to My App" focused="false" />
<Button id="loginButton" text="Log In" focused="true" />
<Poster id="heroImage" uri="https://..." />
</Scene>This XML shows you:
Tags: Component types (Label, Button, Poster, etc.)
Attributes: Properties like id, text, focused, visible
Hierarchy: Nested structure
Now you can write acceptance criteria tests:
Verify that a Label with text "Welcome to My App" is present on screenVerify that a Button with id "loginButton" and focused=true is presentVerify that the login button text says "Log In"Navigate using these keys: ["Down", "Down", "Select"]
Then verify that a Label with text "Account Settings" appearsRun this acceptance test:
Test: Login Flow
Steps:
1. Press Select to click login button
2. Verify keyboard screen appears (look for Label with text "Enter Email")
3. Send text "test@example.com"
4. Press Down to go to password field
5. Send text "password123"
6. Press Select to submit
7. Verify success message appears
8. Take screenshotcreate_webdriver_sessionCreates a new WebDriver session.
Parameters:
webdriver_url (optional): URL of WebDriver server (default: http://localhost:9000)
roku_ip (required): IP address of Roku device
app (optional): App ID or "dev" for sideloaded (default: "dev")
Example:
Create a WebDriver session for Roku at 192.168.1.100end_webdriver_sessionEnds the current session and cleans up.
get_ui_sourceGet the current UI hierarchy as XML or JSON.
Parameters:
parsed (optional): Return JSON instead of XML
Example:
Get the UI source to see current screen structureReturns:
<Scene>
<LayoutGroup id="mainLayout">
<Label id="title" text="Home Screen" />
<Button id="playButton" text="Play" focused="true" />
</LayoutGroup>
</Scene>This is essential for understanding what elements exist and how to query them.
find_elementSearch for a specific element on screen.
Parameters:
text (optional): Text content to match
tag (optional): SceneGraph component type
attributes (optional): Attribute key-value pairs
Examples:
Find a Label with text "Home Screen"Find a Button with id "playButton"Find an element with tag "Poster" and attribute uri="https://example.com/image.jpg"verify_element_presentCheck if an element exists (returns true/false).
Parameters:
Same as find_element
timeout_ms (optional): How long to wait (default: 10000)
Examples:
Verify a Label with text "Loading..." is presentCheck if login button is focused: Button with focused=trueverify_screen_loadedWait for a specific screen to fully load.
Parameters:
screen_marker: Element query that identifies the screen
timeout_ms (optional): Maximum wait time
Example:
Verify the home screen loaded by checking for Label with text "Featured Content"press_keyPress a single remote button.
Parameters:
key: Button name (Home, Back, Up, Down, Left, Right, Select, Play, Pause, etc.)
navigateExecute a sequence of button presses.
Parameters:
keys: Array of keys to press
delay_ms (optional): Delay between presses
Example:
Navigate: Down, Down, Right, Select with 750ms delayssend_textSend text input (for keyboards/forms).
Parameters:
text: Text to type
launch_appLaunch an app with optional deep linking.
Parameters:
app_id: App ID ("dev" for sideloaded)
content_id (optional): Deep link content ID
media_type (optional): Type (movie, series, etc.)
get_player_stateGet current playback state.
Returns: Position, duration, state, buffering info
get_installed_appsList all installed apps.
take_screenshotCapture current screen.
Parameters:
save_path (optional): Where to save the image
Example:
Take a screenshot and save to /tmp/login_screen.pngrun_acceptance_testExecute a complete test case with multiple steps.
Parameters:
test_name: Name of the test
steps: Array of test steps
Step types:
navigate: Execute key sequence
verify_element: Check element is present
press_key: Press single key
send_text: Type text
wait: Pause for duration
screenshot: Capture screen
Example:
Run this acceptance test:
Name: "Video Playback Test"
Steps:
1. Action: navigate, Keys: ["Down", "Down", "Select"], Description: "Select first video"
2. Action: verify_element, Query: {tag: "Video", attributes: {state: "playing"}}, Description: "Verify video is playing"
3. Action: wait, Duration: 5000, Description: "Let video play for 5 seconds"
4. Action: press_key, Key: "Pause", Description: "Pause playback"
5. Action: verify_element, Query: {tag: "Video", attributes: {state: "paused"}}, Description: "Verify video paused"
6. Action: screenshot, Description: "Capture paused state"Elements are queried using combinations of:
{ "text": "Log In" }Finds elements containing this exact text.
{ "tag": "Button" }Finds elements of this SceneGraph type.
Common tags:
Label - Text display
Button - Interactive button
Poster - Image
Video - Video player
LayoutGroup - Container
RowList - Scrollable list
Grid - Grid layout
{
"attributes": {
"id": "loginButton",
"focused": "true"
}
}Common attributes:
id - Unique identifier
focused - Has focus (true/false)
visible - Is visible (true/false)
text - Text content
uri - Image/video URI
{
"tag": "Button",
"text": "Log In",
"attributes": {
"focused": "true"
}
}Finds a Button with text "Log In" that currently has focus.
1. Create WebDriver session for Roku at 192.168.1.100
2. Get UI source to inspect elements
3. Verify these elements are present:
- Label with text "Sign In"
- Button with text "Email Login"
- Button with text "Guest Mode"
4. Take screenshot for documentationRun this acceptance test:
Name: "Video Playback Verification"
Steps:
1. Navigate to content: ["Down", "Down", "Select"]
2. Verify video player loaded: tag=Video
3. Wait 3 seconds for playback to start
4. Verify video is playing: tag=Video, attributes={state: "playing"}
5. Press "Info" to show controls
6. Verify play/pause button visible: tag=Button, text="Pause"
7. Take screenshot of player controlsTest search feature:
1. Press "Search" key
2. Verify keyboard screen: Label with text "Search"
3. Send text "Breaking Bad"
4. Press "Select" to submit
5. Verify results loaded: Label with text "Results for: Breaking Bad"
6. Verify at least one result: tag=Poster (poster images indicate results)Navigate to settings and verify:
1. Press Home
2. Navigate: ["Down", "Down", "Down", "Right", "Right", "Select"]
3. Verify settings screen: Label with text "Settings"
4. Navigate: ["Down", "Select"]
5. Verify account screen: Label with text "Account Information"
6. Get UI source to document screen structureYou must call create_webdriver_session before any other commands.
Ensure the Roku WebDriver server is running:
./RokuWebDriverGet the UI source first: get_ui_source
Inspect the actual XML structure
Adjust your query to match actual elements
Check spelling and capitalization (XML is case-sensitive)
Only works with sideloaded channels or channels packaged on that device
Production channels block source access (security feature)
Make sure your channel is running (not on Home screen)
Screenshots only work when:
Your sideloaded channel is active
Developer mode is enabled
WebDriver has proper access
WebDriver queries can take 500ms-2s depending on complexity
Use specific queries (tag + attributes) for faster results
Avoid overly broad queries
Sideloaded channels only - Production channels block UI introspection
SceneGraph only - Legacy BrightScript channels not supported
No visual comparisons - You get XML structure, not rendered pixels
Element bounds are relative - Absolute position checking is complex
No direct element interaction - You still navigate with D-pad, not clicks
Get UI source → Understand structure → Write queries❌ { "text": "Play" } // Might match multiple elements
✅ { "tag": "Button", "id": "mainPlayButton", "text": "Play" }verify_screen_loaded with appropriate timeoutBreak complex flows into individual test cases.
Document state before/after critical steps.
For multi-step scenarios, use run_acceptance_test to get structured results.
This MCP server can be integrated into CI/CD pipelines:
Provision Roku devices in your test lab
Start WebDriver server on each
Run tests via MCP server
Collect results and screenshots
Publish test reports
RokuHarness is built on the open MCP standard and works with any MCP client. Here are integration examples:
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
# Connect to RokuHarness MCP server
server_params = StdioServerParameters(
command="node",
args=["/path/to/rokuharness-mcp/build/index.js"]
)
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Create WebDriver session
result = await session.call_tool(
"create_webdriver_session",
arguments={
"roku_ip": "192.168.1.100",
"app": "dev"
}
)
# Get UI source
ui_source = await session.call_tool("get_ui_source", {})
# Verify element
verify_result = await session.call_tool(
"verify_element_present",
arguments={
"text": "Welcome",
"tag": "Label"
}
)
# Assert in your test framework
assert verify_result["present"] == True
# Run acceptance test
test_result = await session.call_tool(
"run_acceptance_test",
arguments={
"test_name": "Login Flow",
"steps": [
{
"action": "navigate",
"description": "Go to login",
"keys": ["Down", "Down", "Select"]
},
{
"action": "verify_element",
"description": "Check login screen",
"element_query": {"text": "Sign In"}
}
]
}
)
print(f"Test Status: {test_result['summary']['status']}")import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
const transport = new StdioClientTransport({
command: 'node',
args: ['/path/to/rokuharness-mcp/build/index.js']
});
const client = new Client({
name: 'roku-test-runner',
version: '1.0.0'
}, {
capabilities: {}
});
await client.connect(transport);
// Create session
const session = await client.request({
method: 'tools/call',
params: {
name: 'create_webdriver_session',
arguments: {
roku_ip: '192.168.1.100',
app: 'dev'
}
}
});
// Verify element
const result = await client.request({
method: 'tools/call',
params: {
name: 'verify_element_present',
arguments: {
tag: 'Button',
text: 'Play'
}
}
});
console.log('Element present:', result.present);✅ CI/CD Pipelines - Jenkins, GitHub Actions, GitLab CI, CircleCI
✅ Test Frameworks - Jest, Mocha, Pytest, JUnit
✅ QA Platforms - TestRail, Zephyr, qTest, Xray
✅ Custom Dashboards - Build your own test runner UI
✅ Scheduled Testing - Cron jobs, AWS Lambda, Azure Functions
✅ Any MCP-compatible tool - The protocol is completely open
name: Roku UI Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '20'
- name: Install dependencies
run: |
cd rokuharness-mcp
npm install
npm run build
- name: Start Roku WebDriver Server
run: |
wget https://github.com/rokudev/automated-channel-testing/releases/download/v1.0/RokuWebDriver
chmod +x RokuWebDriver
./RokuWebDriver &
- name: Run tests
run: python tests/run_roku_tests.py
env:
ROKU_IP: ${{ secrets.ROKU_IP }}
- name: Upload screenshots
if: always()
uses: actions/upload-artifact@v3
with:
name: test-screenshots
path: screenshots/This: Natural language via Claude, MCP protocol
Robot: Keyword-driven, separate test files
This: Direct WebDriver access, simpler setup
Appium: Appium ecosystem integration, more tooling
This: Automated, repeatable, fast
Manual: Comprehensive but slow, expensive
For issues or questions:
Roku WebDriver: Roku Community Forums
This MCP Server: Open an issue in this repository
Claude/MCP: Anthropic Documentation
MIT License - Free to use for your Roku testing needs.
Create a new Roku WebDriver session. This must be called before any other commands. Requires the Roku WebDriver server to be running.
| Name | Required | Description | Default |
|---|---|---|---|
| app | No | Optional: App ID or path to sideload. Use "dev" for sideloaded channel. | dev |
| roku_ip | Yes | IP address of the Roku device | |
| webdriver_url | No | URL of the Roku WebDriver server (e.g., http://localhost:9000) | http://localhost:9000 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that a session is created and requires the server to be running, and it notes the ordering constraint. However, it does not describe what the session represents, whether it is destructive, idempotent, or what happens on failure. This leaves gaps in understanding the operational impact.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and front-loaded: the first sentence states the action, and the second adds usage guidance and prerequisites. Every word earns its place, with no redundant content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, but there is no output schema, so the description would need to explain the return value or session behavior. It does not mention what the session returns or how it is referenced later, though it implies it is needed for subsequent commands. This gap prevents a higher score.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all parameters are already well-documented in the schema. The description adds no additional parameter semantics, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function with a specific verb ('Create') and resource ('Roku WebDriver session'). It also distinguishes this tool from siblings by noting it must be called before any other commands, making its role as the session initializer explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use guidance: it must be called before any other commands. It also states the prerequisite that the Roku WebDriver server must be running, which is essential context. There are no alternative tools for this purpose, so exclusions are unnecessary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
End the current Roku WebDriver session and clean up resources.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full transparency burden. It adds 'clean up resources' as a behavioral detail, but does not disclose side effects such as session invalidation or behavior when no active session exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no redundant information. It is well-structured and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a simple tool with no parameters and no output schema. The description adequately covers its purpose and cleanup behavior, making it sufficiently complete for an agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the description does not need to add parameter details. The baseline for zero-parameter tools is 4, which applies here.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'End' and resource 'Roku WebDriver session', clearly stating the tool's purpose. It distinguishes from the sibling tool 'create_webdriver_session' by being the inverse operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly implies when to use the tool (to end a session), but does not explicitly mention when not to use it or name alternatives. Context is clear and no exclusions are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Find a UI element on screen using text, tag, and/or attributes. Returns element details if found.
| Name | Required | Description | Default |
|---|---|---|---|
| tag | No | SceneGraph component type (e.g., "Label", "Button", "Poster") | |
| text | No | Text content to search for | |
| attributes | No | Element attributes to match (e.g., {"id": "loginButton", "focused": "true"}) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must fully disclose behavior. It only states "Returns element details if found," which omits critical behaviors such as what happens when not found (null? error?), whether it searches the entire visible screen, how matching works (exact/partial), or if it waits for the element. This leaves significant behavioral ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one concise sentence that includes both the action and the return value. No filler or repetition, every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's role and the absence of an output schema and annotations, the description is under-specified. It does not explain the shape of returned element details, failure semantics, or matching rules. For an agent to invoke this tool reliably, more behavioral context is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes all three parameters fully. The description adds "and/or", suggesting combinability, but does not clarify whether any parameter is required or how attributes are matched (exact string comparison). With 100% schema coverage, the description adds marginal value beyond the schema, so a baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"Find a UI element on screen using text, tag, and/or attributes" clearly states the verb and resource. It also notes that it returns element details, which differentiates it from verify_element_present (likely a presence check). The purpose is unambiguous and distinguishes this tool from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this tool is used when you need to locate a UI element by any of the given criteria. However, it does not explicitly mention when not to use it or name alternatives like verify_element_present or get_ui_source. Usage is implied but not explicitly guided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get list of all installed apps on the device.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It implies a read-only operation via the verb 'get' but does not explicitly state safety, return format, or whether system apps are included. No additional behavioral context is provided.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence that front-loads the purpose without unnecessary words. Every word contributes to the meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (no params, no output schema), the description is adequate but could be more explicit about what the returned list contains (e.g., app names, package IDs). It is not fully complete but sufficient for a straightforward query.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has no parameters, so the baseline of 4 applies. There is no parameter information to add, and the description does not need to compensate for schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Get') and resource ('list of all installed apps') with a scope ('on the device'). It distinguishes itself from sibling tools like launch_app or get_ui_source, which serve different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives, no exclusions, and no context about use cases. It simply states what the tool does without helping the agent decide among the sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get current media player state (position, duration, state, buffering info).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully disclose behavioral traits. It does list the return contents (position, duration, state, buffering info), which provides some transparency. However, it does not mention side effects (or lack thereof), assumptions about a running media player, potential errors, or output format, leaving the agent with only partial insight.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence: 'Get current media player state (position, duration, state, buffering info).' It includes the verb, resource, and key data points with no redundancy or unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the core function and return fields, but it does not provide context about the required environment (e.g., media player must be running), return value types or units, or when this tool is applicable relative to the UI automation workflow. Given no annotations and no output schema, more context would improve completeness, but the basic getter function is adequately described.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has 0 parameters, so there are no parameter semantics to clarify. Per the rubric, a 0-parameter tool receives a baseline of 4. The description appropriately omits parameter information since none exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Get current media player state' and specifies the exact data returned (position, duration, state, buffering info). This is a specific verb+resource and is distinct from all sibling tools, which focus on UI automation elements like webdriver sessions and element finding.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus any alternative. It simply states what it does, with no mention of context, prerequisites, or exclusions. No sibling tool is referenced, so the agent is left without explicit usage direction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get the current UI hierarchy as SceneGraph XML. Essential for understanding what elements are on screen and writing element queries.
| Name | Required | Description | Default |
|---|---|---|---|
| parsed | No | If true, return parsed JSON structure instead of raw XML |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It correctly implies a read-only getter operation via 'Get,' and adds the useful context that it's foundational for element queries. However, it doesn't mention potential edge cases or output size, leaving some gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler: the first states the core behavior, the second explains why it matters. It is front-loaded and efficiently worded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one optional parameter and no output schema, the description covers the primary return type (SceneGraph XML) and its main use case (writing element queries). It is sufficiently complete without needing to restate the parameter documentation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% because the 'parsed' parameter has a clear description. The tool description adds no extra semantic meaning to the parameter, but none is needed since the schema already explains its effect on output format.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (get), the resource (current UI hierarchy), and the format (SceneGraph XML). It also emphasizes its role in understanding screen elements and writing queries, which distinguishes it from siblings like take_screenshot and find_element.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description says the tool is 'essential for understanding what elements are on screen and writing element queries,' providing clear context for when to use it. It does not explicitly name alternatives or exclusions, but the implied usage is strong enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Launch a specific app with optional deep linking parameters.
| Name | Required | Description | Default |
|---|---|---|---|
| app_id | Yes | App ID to launch (e.g., "12" for Netflix, "dev" for sideloaded) | |
| content_id | No | Optional content ID for deep linking | |
| media_type | No | Optional media type (e.g., "movie", "series") |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It only names the action without stating side effects, requirements, return values, or failure behavior (e.g., what happens if the app is not installed).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The single-sentence description is front-loaded and contains no fluff. However, it is extremely brief and omits details that could be included without hurting conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a three-parameter tool with no output schema and no annotations, this description is under-specified. It does not explain return values, prerequisites, or behavioral outcomes, leaving the agent with significant ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides 100% coverage with descriptions for each parameter, including examples. The description adds minimal context by labeling content_id and media_type as 'deep linking parameters,' which is a baseline contribution.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Launch') and the resource ('a specific app'), distinguishing it from sibling tools like get_installed_apps or navigate. The verb+resource is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It does not mention prerequisites (e.g., an active webdriver session) or exclusions, leaving the agent to infer use cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Press a remote control button. Common keys: Home, Back, Up, Down, Left, Right, Select, Play, Pause.
| Name | Required | Description | Default |
|---|---|---|---|
| key | Yes | Key to press |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It only states what action is performed, but does not disclose side effects, whether the press is synchronous, what happens on invalid keys, or any post-press behavior. This leaves the agent guessing about important runtime behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence followed by a useful list of common keys. It is front-loaded with the primary action and contains no filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter action, the description is reasonably complete: it states what the tool does and provides example values. However, it does not mention timing or synchronization behavior (e.g., whether it waits for the resulting UI state), which could be relevant in an automation context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers the parameter with a generic 'Key to press' description. The tool description adds meaningful context by listing common accepted keys (Home, Back, etc.), which clarifies expected values beyond the schema's minimal description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: 'Press a remote control button.' It clearly distinguishes from sibling tools like navigate (which implies navigation) and send_text (which sends text input). The list of common keys further clarifies the tool's domain.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when a remote control button needs to be pressed ('Press a remote control button'), but it does not explicitly state when to avoid this tool or provide alternatives. No exclusions or comparisons to sibling tools are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a complete acceptance test with multiple verification steps. Returns detailed pass/fail results.
| Name | Required | Description | Default |
|---|---|---|---|
| steps | Yes | Array of test steps to execute | |
| test_name | Yes | Name of the test case |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must disclose behavioral traits on its own. It mentions that it returns pass/fail results, but it does not disclose potential side effects (e.g., UI interactions that may modify state), session requirements, or failure behavior. This is insufficient for a tool that executes multi-step actions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that conveys the core purpose and result type without any redundant or filler content. It earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The schema richly defines inputs, but there is no output schema and the description only vaguely mentions 'detailed pass/fail results.' It lacks information about the result structure, prerequisites (e.g., a webdriver session), or edge-case behavior. This makes it minimally viable but not comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage for parameters, so the description does not need to add detailed parameter info. The phrase 'multiple verification steps' loosely maps to the 'steps' array, but the schema already fully explains each action type. Baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action ('Run') on a specific resource ('complete acceptance test') and distinguishes it from sibling tools by noting 'multiple verification steps' and 'detailed pass/fail results.' This makes it evident that this is a high-level aggregate test runner rather than a single UI operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for multi-step acceptance testing, but it does not explicitly state when to use this tool versus the individual action tools (e.g., press_key, navigate, verify_element_present). No exclusions or alternative guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Send text input (for search fields, forms, etc.).
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | Text to send |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It only states the action and purpose but does not mention whether the text replaces existing content, appends, or requires an element to be focused. This leaves significant behavioral ambiguity for an agent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and context. There is no extraneous information, making it appropriately sized and easily parsed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a single parameter, no output schema, and no annotations, so the description alone must convey all necessary context. It provides the basic purpose but omits any details about expected behavior or side effects, leaving it slightly incomplete for a fully autonomous agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage, with the parameter 'text' already described as 'Text to send'. The tool description adds no further semantic detail about the parameter, relying entirely on the schema. This meets the baseline for schema-backed parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Send text input') and provides context ('for search fields, forms, etc.'). This identifies the tool's purpose as entering text, but it does not explicitly distinguish it from sibling tools like press_key, which also handles keyboard input.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'for search fields, forms, etc.' gives clear context for when to use this tool, implying it is for text entry in UI fields. However, it does not explicitly mention when not to use it or list alternative tools, though the context is sufficient for a basic understanding.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Capture a screenshot of the current screen state. Returns base64-encoded PNG.
| Name | Required | Description | Default |
|---|---|---|---|
| save_path | No | Optional: Path to save screenshot file |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses the return format (base64-encoded PNG) but omits other behavioral traits such as whether saving via save_path is alternative to returning, whether it requires an existing session, or potential side effects. The optional save_path introduces ambiguity about the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with the main action. No fluff or redundancy. Every word contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, and the description covers the core function and return type. However, the behavior of save_path is ambiguous (does it save, return, or both?), and no context about session requirements or whether the screenshot is of the entire screen or viewport is given. This leaves gaps for a complete understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (save_path is documented). The description adds no extra meaning about the parameter or how it interacts with the return value. Baseline of 3 applies due to full schema coverage, and no misleading or confusing param info is present.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Capture a screenshot') and the target ('current screen state'). It distinguishes itself from sibling tools like get_ui_source and verify_screen_loaded by specifying a visual capture and its return format.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No usage context is provided. There is no guidance on when to use this tool versus alternatives like get_ui_source, nor any mention of prerequisites (e.g., an active session). The description only states what it does, not when to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Verify that a specific element is present on screen. Returns true/false. Critical for acceptance criteria validation.
| Name | Required | Description | Default |
|---|---|---|---|
| tag | No | SceneGraph component type | |
| text | No | Text content to search for | |
| attributes | No | Element attributes to match | |
| timeout_ms | No | How long to wait for element to appear (default: 10000) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the burden of behavioral disclosure. It mentions 'Returns true/false,' which is helpful, but it fails to mention waiting behavior, timeout semantics, or whether the tool is read-only. It does not disclose how matching works or what happens when the element is not found. This is a significant gap for a verification tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, consisting of three short sentences. It front-loads the core purpose ('Verify that a specific element is present on screen'), states the return type, and adds a contextual note. The final sentence about acceptance criteria is somewhat filler but not harmful. Overall, it is efficient without unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has four parameters, one being a nested object, and no output schema. The description is too brief to be complete: it does not explain how the element is matched, what 'present' means in terms of waiting (e.g., does it wait for timeout_ms?), or what the return value represents in edge cases. The description lacks the depth needed for a tool with this complexity and no annotations or output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides 100% coverage of the four parameters with descriptions. The tool description adds no additional parameter semantics; it only says 'specific element' without elaborating on tag, text, attributes, or timeout. Since schema coverage is high, the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Verify that a specific element is present on screen.' It uses a specific verb ('Verify') and resource ('element'), and distinguishes itself from siblings like find_element by noting it 'Returns true/false', indicating a boolean verification result rather than returning the element itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides some usage context by stating 'Critical for acceptance criteria validation,' implying it is used for assertions. However, it does not explicitly contrast with alternatives like find_element or verify_screen_loaded, nor does it state when not to use this tool. The guidance is implicit rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Wait for a specific screen to load by checking for a marker element. Returns true when screen loads or false on timeout.
| Name | Required | Description | Default |
|---|---|---|---|
| timeout_ms | No | Maximum time to wait for screen (default: 10000) | |
| screen_marker | Yes | Element query that uniquely identifies the screen |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of disclosure. It clearly states the blocking wait behavior and the return value on both success and timeout ('Returns true when screen loads or false on timeout'). This is useful behavioral context beyond the schema, though it does not detail polling behavior or error conditions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the primary action, and contains no redundant or vague wording. Every word contributes to understanding what the tool does.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a relatively simple tool, the description covers the essential behavior. However, without an output schema or annotations, it could say more about edge cases (e.g., invalid selectors), what constitutes 'loaded' (visible vs. present), and how it relates to sibling verification tools. The lack of this context leaves some ambiguity for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides descriptions for both parameters: timeout_ms ('Maximum time to wait for screen') and screen_marker ('Element query that uniquely identifies the screen'). The description adds no additional parameter-level meaning, so with 100% schema coverage, a baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action: 'Wait for a specific screen to load by checking for a marker element.' This distinguishes it from sibling tools like verify_element_present, which focuses on a single element rather than a screen-level condition.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when you need to ensure a screen is ready before proceeding, but it provides no explicit guidance on when to use this tool versus alternatives like verify_element_present or find_element. No comparisons or exclusion criteria are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.0.0create_webdriver_sessionend_webdriver_sessionfind_elementget_installed_appsget_player_stateget_ui_sourcelaunch_appnavigatepress_keyrun_acceptance_testsend_texttake_screenshotverify_element_presentverify_screen_loadedScored across 14 tools
Most tools have clearly distinct purposes, but there is some overlap among find_element, verify_element_present, and verify_screen_loaded since they all deal with UI state. Descriptions clarify their different return types and use cases.
All tool names follow a consistent verb_noun pattern in snake_case (e.g., create_webdriver_session, press_key, verify_element_present). No mixing of conventions.
14 tools is well-scoped for a device automation server, covering session lifecycle, UI interaction, verification, and media state without being excessive.
The tool set covers the full lifecycle of device testing: session management, app control, UI inspection and interaction, verification, screenshots, and media state. No critical gaps for the stated purpose.
Drive real Android & iOS devices and web browsers from natural language for mobile + web QA. 290+ tools across device control, app management, automation sessions, browser automation, and flow recording / replay. Bearer-auth — get a token at robotactions.com → Profile → API Tokens.
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
Katalon True Platform MCP: design tests, run with AI, upload reports, assess release readiness.
Browser-based QA for AI-built software. Test pages with real browsers via agents.
