idb-ui-describe

🔍 Query UI accessibility tree - discover tappable elements and text fields for precise automation

What it does

Queries iOS accessibility tree to discover UI elements, their properties (type, label, enabled state), coordinates (frame, centerX, centerY), and accessibility identifiers. Returns full tree with progressive disclosure (summary + cache ID for full data), element-at-point queries for tap validation, and data quality assessment (rich/moderate/minimal) to guide automation strategy. Automatically parses NDJSON output to extract all elements (not just first), includes AXFrame coordinate parsing for precise tapping, and caches large outputs to prevent token overflow.

Progressive Filtering: Supports 4 filter levels for element discovery - start conservative with moderate filtering (default), escalate to permissive/none if minimal data found.

iOS Compatibility: Recognizes iOS-specific accessibility fields (role, role_description, AXLabel, AXFrame) in addition to standard fields.

Why you'd use it

• Discover all tappable elements from accessibility tree - buttons, cells, links identified by JSON element objects

• Get precise tap coordinates (centerX, centerY) for elements without needing screenshots

• Assess data quality before choosing automation approach - rich data enables precise targeting, minimal data requires screenshots

• Validate tap coordinates by querying elements at specific points before execution

• Progressive disclosure prevents token overflow on complex UIs - get summary first, full tree on demand

• Progressive filter escalation - start with moderate filtering, escalate to permissive/none if minimal data found

Parameters

Required

• operation (string): "all" | "point"

Point operation parameters

• x (number, required for point operation): X coordinate to query

• y (number, required for point operation): Y coordinate to query

Optional

• udid (string): Target identifier - auto-detects if omitted

• screenContext (string): Screen name for context (e.g., "LoginScreen")

• purposeDescription (string): Query purpose (e.g., "Find tappable button")

• filterLevel (string): "strict" | "moderate" | "permissive" | "none" (default: "moderate")

  • strict: Only obvious interactive elements via type field (original behavior)

  • moderate: Include iOS roles (role, role_description) - DEFAULT, fixes iOS button detection

  • permissive: Any element with role/type/label information

  • none: Return everything (debugging)

Returns

For "all": UI tree summary with element counts (total, tappable, text fields), data quality assessment (rich/moderate/minimal), top 20 interactive elements preview with centerX/centerY coordinates, uiTreeId for full tree retrieval, current filter level, and guidance on automation strategy including suggestions to escalate filter level if minimal data found.

For "point": Element details at coordinates including type, label, value, identifier, frame coordinates (x, y, centerX, centerY), enabled state, and tappability.

Examples

Query full UI tree with default moderate filtering

const result = await idbUiDescribeTool({
  operation: 'all',
  screenContext: 'LoginScreen',
  purposeDescription: 'Find email and password fields'
});
// Result includes elements with centerX, centerY for direct tapping

Progressive filter escalation pattern

// 1. Start with default (moderate)
let result = await idbUiDescribeTool({ operation: 'all' });

// 2. If minimal data, try permissive
if (result.summary.dataQuality === 'minimal') {
  result = await idbUiDescribeTool({
    operation: 'all',
    filterLevel: 'permissive'
  });
}

// 3. If still minimal, try none (return everything)
if (result.summary.dataQuality === 'minimal') {
  result = await idbUiDescribeTool({
    operation: 'all',
    filterLevel: 'none'
  });
}

// 4. If STILL minimal, fall back to screenshots
if (result.summary.dataQuality === 'minimal') {
  // Use screenshot-based approach
}

Validate element at tap coordinates

const element = await idbUiDescribeTool({
  operation: 'point',
  x: 200,
  y: 400
});
// Element includes frame coordinates if available

Related Tools

• idb-ui-tap: Tap discovered elements using centerX/centerY coordinates

• screenshot: Capture screenshot for visual element identification

• idb-ui-find-element: Semantic element search by label/identifier

• accessibility-quality-check: Quick assessment before choosing approach