Skip to main content
Glama
dirkknibbe

UI Perception Engine

by dirkknibbe
OverviewSchemaRelated ServersScoreDiscussions
TypeScript
Local

UI Perception Engine

A perception layer for AI agents — human-like understanding of web interfaces.

Fuses structural (DOM + a11y tree + CSS), visual (three-tier vision pipeline), and temporal data into a unified UI Scene Graph that an LLM can reason over fluently. Exposed as an MCP server so Claude can navigate and inspect any live URL.

What It Does

  • Navigates to any URL in a real Playwright browser

  • Extracts a compact, LLM-readable scene graph from the live DOM

  • Detects UI elements visually via OmniParser V2, understands layout via Qwen3-VL, and performs deep UX analysis via Claude Vision

  • Tracks UI transitions and diffs between states

  • Predicts affordances (what you can interact with, and what happens when you do)

  • Exposes everything as MCP tools that Claude can call directly

Related MCP server: MCP Browser Use Server

MCP Tools

Tool

Description

navigate

Navigate to a URL and return the initial scene graph

get_scene

Re-capture the current scene (compact text or full JSON)

get_affordances

List interactive elements ranked by priority

act

Execute browser actions: click, type, scroll, hover, keypress, navigate, wait

get_console_logs

Return captured browser console messages

get_network_errors

Return failed network requests

get_screenshot

Capture a screenshot and return it as an image

detect_elements

Run element detection (OmniParser or Claude Vision fallback)

analyze_visual

Visual understanding via Qwen3-VL: hierarchy, contrast, spacing, UX

compare_states

Diff current vs previous scene graph — shows what changed

watch

Start real-time keyframe capture (CDP screencast + perceptual hashing)

stop_watch

Stop keyframe capture and return summary of changes

Three-tier vision pipeline (visual=true)

Both navigate and get_scene accept an optional visual: true parameter. When enabled, the engine runs a three-tier vision pipeline:

  • Tier A: OmniParser V2 — Fast element detection (~0.8s). A YOLOv8 + Florence-2 model running as a Python sidecar on port 8100. Detects buttons, inputs, images, icons, and other UI elements with bounding boxes and labels.

  • Tier B: Qwen3-VL via Ollama — Visual understanding (~2-4s). Analyzes the screenshot with detected element context to assess visual hierarchy, contrast issues, spacing problems, affordance clarity, and state indicators.

  • Tier C: Claude Vision API — Deep UX analysis (~3-5s, on-demand only). Provides detailed qualitative analysis when requested via analyze_visual or depth: 'deep'.

Graceful degradation: Each tier skips silently if its backing service is unavailable. The system works with any combination of services running — from all three tiers down to structural-only analysis with no vision services at all.

Use detect_elements for fast Tier A detection only, or analyze_visual for the full Tier A + B pipeline.

act action types

Type

Parameters

click

x, y

clickSelector

selector

type

text, selector (optional)

scroll

direction (up/down), amount (optional)

hover

x, y

wait

ms

navigate

url

back

—

pressKey

key

Installation

From npm

npm install -g ui-perception-engine

Or use without installing:

npx ui-perception-engine

Install Playwright browsers (first time only):

npx playwright install chromium

From source

git clone https://github.com/dirkknibbe/uipe.git
cd uipe
pnpm install
pnpm build

Claude Code MCP Configuration

Add to your Claude Code MCP config (~/.claude/mcp.json or project .mcp.json):

{
  "mcpServers": {
    "ui-perception-engine": {
      "command": "npx",
      "args": ["ui-perception-engine"]
    }
  }
}

Or if installed globally:

{
  "mcpServers": {
    "ui-perception-engine": {
      "command": "uipe"
    }
  }
}

Or from a local clone:

{
  "mcpServers": {
    "ui-perception-engine": {
      "command": "node",
      "args": ["/path/to/uipe/ui-perception-engine/packages/core/dist/src/mcp/index.js"],
      "env": {
        "OLLAMA_URL": "http://localhost:11434",
        "OLLAMA_MODEL": "qwen3-vl:8b",
        "OMNIPARSER_URL": "http://localhost:8100",
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

Environment variables

Variable

Purpose

ANTHROPIC_API_KEY

Claude Vision API — detection fallback + deep analysis (Tier C)

OLLAMA_URL

Ollama server URL (default: http://localhost:11434)

OLLAMA_MODEL

Vision model name (default: qwen3-vl:8b)

OMNIPARSER_URL

OmniParser V2 sidecar URL (default: http://localhost:8100)

See .env.example for the full list of configurable variables including frame capture, browser, and temporal settings.

Local Vision Services

The three-tier vision pipeline uses two local services. Both are optional — the system degrades gracefully without them.

Ollama (Tier B — visual understanding):

# Install Ollama: https://ollama.com
ollama pull qwen3-vl:8b
ollama list                    # verify model is available
# Ollama serves on http://localhost:11434 by default

OmniParser V2 (Tier A — element detection):

OmniParser runs as a Python FastAPI sidecar on port 8100. See the Local Vision Handoff doc section 5 for full setup instructions.

# Quick check if OmniParser is running:
curl -s http://localhost:8100/health

Without local services: If neither Ollama nor OmniParser is running, visual=true falls back to Claude Vision API (requires ANTHROPIC_API_KEY). If no vision service is available at all, the engine uses structural-only analysis (DOM + a11y tree).

Using with the live-deployment-check Skill

The live-deployment-check skill pairs directly with this MCP server to visually verify a deployed site or app — catching broken images, empty routes, stuck spinners, and placeholder text that only surface in a real browser.

Workflow

1. navigate(url)            → load the page, get initial scene
2. get_scene()              → re-capture after JS hydrates (critical for SPAs)
3. get_console_logs()       → check for JS errors (type="error")
4. get_network_errors()     → check for failed API/resource requests
5. Scan scene output        → look for broken signals (see below)
6. act() on nav links       → walk routes, verify each one loads
7. Report findings          → list what's working and what's broken

Common Signals in Scene Output

# Broken image:
img[img]:"broken"

# Empty SPA route (component failed to load):
router-outlet[element]          ← no children = problem

# Stuck loading spinner:
progressbar[progressbar]        ← present after JS settles = API error

# Route loaded correctly:
router-outlet[element]
  app-order-list[element]:"Order Management..."   ← has content = good

Example

// After deploying an Angular app
navigate("http://your-app.vercel.app")
get_scene()                     // wait for hydration
→ check router-outlet has content, no broken img nodes

// Walk routes
act({ type: "clickSelector", selector: "a[href='/orders']" })
get_scene()
→ verify orders page loaded

act({ type: "clickSelector", selector: "a[href='/customers']" })
get_scene()
→ verify customers page loaded

What to Check

  • Broken images — img nodes with "broken" content

  • Empty routes — router-outlet with no child elements

  • Stuck spinners — progressbar still present after get_scene()

  • Placeholder text — undefined, null, TODO, <repo-url> in visible text

  • Error pages — 404 or error component rendered instead of expected content

Development

# Root workspace (delegates to all packages via -r)
pnpm test          # run all tests
pnpm build         # compile TypeScript
pnpm lint          # lint all packages

# @uipe/core package only
pnpm -F @uipe/core test:watch          # watch mode
pnpm -F @uipe/core mcp                 # start MCP server (after build)
pnpm -F @uipe/core start:dev           # check services + start MCP server
pnpm -F @uipe/core exec vitest run --reporter=verbose  # verbose test output

Architecture

packages/
├── contracts/          ← shared types (@uipe/contracts)
└── core/               ← perception engine + MCP server (@uipe/core)
    └── src/
        ├── config.ts           ← centralized config (dotenv)
        ├── types/              ← internal types
        ├── browser/            ← BrowserRuntime (Playwright)
        ├── pipelines/
        │   ├── structural/     ← DOM + a11y tree extraction
        │   ├── visual/
        │   │   ├── index.ts        ← Three-tier orchestrator (detect/understand/deep)
        │   │   ├── omniparser.ts   ← OmniParser V2 client (Tier A)
        │   │   ├── claude-vision.ts ← Claude Vision API (Tier C)
        │   │   ├── ollama-vision.ts ← Qwen3-VL via Ollama (Tier B)
        │   │   └── frame-capture.ts ← CDP screencast + perceptual hashing
        │   ├── fusion/         ← merge visual + structural → SceneGraph
        │   ├── temporal/       ← change detection + state tracking
        │   └── affordance/     ← predict interaction outcomes
        ├── mcp/                ← MCP server (12 tools)
        └── utils/

Viewport default: 1280x720 (configurable via env) Screenshot format: PNG (lossless, required for vision models)

This server cannot be installed

F
license - not found
-
quality - not tested
C
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity
Issues opened vs closed

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/dirkknibbe/uipe'

If you have feedback or need assistance with the MCP directory API, please join our Discord server