Skip to main content
Glama

run_test_scenario

Execute ordered test scenarios on Android devices with multiple steps and assertions to verify app functionality through automated actions and result checks.

Instructions

Execute an ordered test scenario with multiple steps and assertions. Each step performs an action and optionally verifies the result. Returns detailed pass/fail results per step.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
nameYesName of the test scenario
stepsYesOrdered list of test steps
stop_on_failureNo
device_idNoDevice serial number

Implementation Reference

  • The core implementation of the runTestScenario function which executes a sequence of test steps.
    export async function runTestScenario(
      scenarioName: string,
      steps: TestStep[],
      options: {
        deviceId?: string;
        stopOnFailure?: boolean;
        captureScreenshotsOnFailure?: boolean;
      } = {}
    ): Promise<TestScenarioResult> {
      const {
        deviceId,
        stopOnFailure = false,
        captureScreenshotsOnFailure = true,
      } = options;
    
      const resolved = await deviceManager.resolveDeviceId(deviceId);
      const scenarioStart = Date.now();
    
      const result: TestScenarioResult = {
        name: scenarioName,
        totalSteps: steps.length,
        passed: 0,
        failed: 0,
        skipped: 0,
        results: [],
        durationMs: 0,
      };
    
      let shouldSkip = false;
    
      for (const step of steps) {
        if (shouldSkip) {
          result.results.push({
            stepName: step.name,
            status: 'skip',
            durationMs: 0,
          });
          result.skipped++;
          continue;
        }
    
        const stepStart = Date.now();
        let status: TestResult['status'] = 'pass';
        let error: string | undefined;
        let screenshot: string | undefined;
    
        try {
          // Execute the action
          await executeTestAction(step.action, step.args, resolved);
    
          // Wait if specified
          if (step.delayAfter) {
            await new Promise(resolve => setTimeout(resolve, step.delayAfter));
          }
    
          // Run assertion if specified
          if (step.assert) {
            await runAssertion(step.assert, resolved);
          }
        } catch (err) {
          status = 'fail';
          error = err instanceof Error ? err.message : String(err);
    
          // Capture screenshot on failure
          if (captureScreenshotsOnFailure) {
            try {
              const sc = await captureScreenshot(resolved);
              screenshot = sc.base64;
            } catch {
              // Ignore screenshot failure
            }
          }
    
          if (stopOnFailure) {
            shouldSkip = true;
          }
        }
    
        const stepResult: TestResult = {
          stepName: step.name,
          status,
          durationMs: Date.now() - stepStart,
          error,
          screenshot,
        };
    
        result.results.push(stepResult);
        if (status === 'pass') result.passed++;
        else result.failed++;
    
        log.info(`Step "${step.name}": ${status}`, {
          durationMs: stepResult.durationMs,
          ...(error ? { error } : {}),
        });
      }
    
      result.durationMs = Date.now() - scenarioStart;
    
      log.info(`Scenario "${scenarioName}" completed`, {
        passed: result.passed,
        failed: result.failed,
        skipped: result.skipped,
        durationMs: result.durationMs,
      });
    
      return result;
    }
  • TypeScript interface defining the TestStep structure.
    export interface TestStep {
      /** Step identifier */
      name: string;
      /** Action to perform */
      action: string;
      /** Action arguments */
      args: Record<string, unknown>;
      /** Optional assertion to verify after action */
      assert?: {
        type: 'element_exists' | 'element_not_exists' | 'text_visible';
        selector?: ElementSelector;
        text?: string;
        timeoutMs?: number;
      };
      /** Delay after action in ms */
      delayAfter?: number;
    }
  • The registration and tool handler invocation for the 'run_test_scenario' MCP tool.
    },
    async ({ name, steps, stop_on_failure, device_id }) => {
      return await metrics.measure('run_test_scenario', device_id || 'default', async () => {
        const result = await runTestScenario(name, steps as TestStep[], {
          deviceId: device_id,
          stopOnFailure: stop_on_failure,
          captureScreenshotsOnFailure: false, // Don't include base64 screenshots in results to save space
        });
        return {
          content: [{
            type: 'text' as const,
            text: JSON.stringify({
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses the return format ('detailed pass/fail results per step') and execution order ('ordered'), but omits critical behavioral details like the stop_on_failure semantics, whether failed assertions halt execution, or side effects on device state.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two well-constructed sentences with zero waste. First sentence front-loads core functionality (execution + assertions), second sentence covers output contract. Appropriate length for the complexity level.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema exists, the description adequately explains return values ('pass/fail results per step'). For a complex orchestration tool with nested step definitions, it covers the essential contract, though mentioning execution flow control (stop_on_failure) would improve completeness given the lack of annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 75% (3 of 4 params described). The description adds valuable semantic context for the complex 'steps' structure, explaining that each step 'performs an action and optionally verifies the result,' which helps interpret the nested action/args/assert structure. Does not compensate for the undocumented stop_on_failure parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific verb (Execute), resource (test scenario), and key capabilities (ordered steps, assertions). It effectively distinguishes this from single-action siblings like tap or click_element by emphasizing 'multiple steps' and 'scenario' orchestration.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance provided on when to use this composite tool versus individual action tools (tap, swipe, click_element) available as siblings. No mention of prerequisites or when to prefer this over replay_recording for automation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/divineDev-dotcom/android_mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server