# Agent Eye

Give AI agents eyes on the web. Agent Eye is an MCP server that wraps Playwright for local browser search, navigation, extraction, and interaction. It features intelligent LLM-based DOM extraction with multi-pass reconciliation, consensus voting, DOM caching, and field-specific confidence thresholds.

## What Is Implemented

- Browser navigation with persistent or incognito modes
- Search, interact, screenshot, content extraction, JS evaluation tools
- **5-Phase LLM-based DOM extraction** (chunking, reconciliation, consensus voting, caching, field thresholds)
- **Goal Planner preamble** — sanitizes the user goal, extracts entities, selects a browsing strategy, and decomposes into atomic sub-tasks before the step loop starts
- **News-gathering hardening** — quality gates block premature completion, URL-confidence filtering prefers real article links, and deterministic recovery (scroll + listing fallback) gathers missing items
- AI workflow orchestration tool powered by Ollama
- Session persistence with cookie save/load per profile
- Domain allowlist/denylist checks
- Memory pressure protection
- .env support via dotenv bootstrap at startup
- Pre-commit git hooks (Husky) for test validation
- AGPL-3.0 license (free for non-commercial/personal use)

## Requirements

- Node >= 22.14.0
- Playwright Chromium installed

## Setup

1. Install dependencies

```bash
npm install
```

1. Install Chromium

```bash
npx playwright install chromium
```

1. Configure environment

```bash
cp .env.sample .env
# or: cp .env.example .env
```

## Run

Development:

```bash
npm run dev
```

Build + start:

```bash
npm run build
npm run start
```

Type check:

```bash
npm run typecheck
```

Tests:

```bash
npm run test
```

## Environment Variables

Core:

- AGENT_EYE_HEADLESS=true|false
- AGENT_EYE_NAV_TIMEOUT_MS=30000
- AGENT_EYE_MAX_CHARS=50000
- AGENT_EYE_PROFILE_ROOT=~/.agent-eye/profiles
- AGENT_EYE_MODEL=deep-seek|llama2|default (content-budget profile for browser_get_content, not Ollama model name)
- AGENT_EYE_SKILL_STALE_FOCUSED_CHARS=400 (refresh skill if cached focused content is too small)
- AGENT_EYE_SKILL_STALE_SELECTOR_MATCHES=1 (refresh skill if cached selectors barely match DOM)
- AGENT_EYE_SKILL_STALE_FAILURE_THRESHOLD=2 (refresh skill after repeated selector failures)

Ollama:

- OLLAMA_BASE_URL=<http://localhost:11434>
- OLLAMA_MODEL=gemma4:e4b-it-q4_K_M
- OLLAMA_NUM_CTX=16384 (shared across all calls)
- OLLAMA_KEEP_ALIVE=10m

Recommended with your local model:

- Keep AGENT_EYE_MODEL=default
- Set OLLAMA_MODEL to your installed model tag (for example gemma4:e4b-it-q4_K_M)
- Keep OLLAMA_NUM_CTX stable for all calls to prevent duplicate model instances in memory
- Use OLLAMA_KEEP_ALIVE=10m (or higher) for smoother multi-step and chunked-answer runs

Note: OLLAMA_MODEL from .env is loaded automatically because src/index.ts imports dotenv/config before server startup.

## MCP Tools

- browser_navigate
- browser_get_content
- browser_search
- browser_interact
- browser_screenshot (with intelligent SPA hydration waits)
- browser_evaluate_js
- browser_analyze_page (with skill cache + stale-skill auto-refresh)
- browser_extract_section (token-efficient targeted extraction)
- browser_extract_structured (LLM-based structured extraction with templates: showtime, news_article, product, search_result, generic)
- browser_ocr_chunk (visual fallback when selector extraction is weak)
- browser_get_network_logs
- ai_orchestrate_workflow
- **ai_plan_goal** — pre-flight goal planner: sanitize → extract entities → select strategy → decompose into sub-tasks

## AI Orchestration

### ai_orchestrate_workflow

Accepts a high-level goal and drives a deterministic YouTube-metadata extraction workflow. Built-in 5-phase LLM extraction:

- **Phase 1**: Chunked per-field extraction with confidence scoring
- **Phase 2**: Global reconciliation pass for conflict resolution
- **Phase 3**: Consensus voting across multiple runs for reliability
- **Phase 4**: DOM fingerprint caching to skip redundant extraction
- **Phase 5**: Field-specific confidence thresholds for domain-aware validation

See [EXTRACTION_PHASES.md](EXTRACTION_PHASES.md) for detailed architecture.

Input: `goal`, `context`, `maxSteps`, `model`  
Output: `success`, `steps_executed`, `history`, `result`, `error`

Use cases: [AI_ORCHESTRATION_USE_CASES.md](AI_ORCHESTRATION_USE_CASES.md)

### ai_plan_goal — Goal Planner

Runs before the browser step loop to reduce per-step LLM pressure on local 9B-class models. Makes three focused LLM calls:

1. **Sanitize** — correct spelling/grammar in the raw goal; extract entities (`movies`, `cinemas`, `cities`, `dateRange`)
2. **Strategy** — choose one of four strategies: `movie-first`, `location-first`, `search-first`, `direct-navigate`
3. **Decompose** — emit ordered `SubTask[]` with `stepBudget`, `successCriteria`, `suggestedTools`

Step budget scale:

| Budget | Meaning                                           |
| ------ | ------------------------------------------------- |
| 1      | Compile / summarize data already in memory        |
| 2      | Navigate to a known URL or single click           |
| 3      | Navigate + analyze page structure                 |
| 4      | Interact with a dynamic widget + read AJAX result |
| 5      | Iterate over 3–5 pages or a complex widget        |
| 6      | Iterate over 6–10 pages (e.g., 10 article visits) |
| 7      | Complex multi-step workflow with retries          |
| 8      | Long iterative data collection (>10 pages)        |

Input: `goal`, `model`, `allowPreflightSearch`, `replanCount`, `completedSubTaskIds`  
Output: `GoalPlan` — `sanitizedGoal`, `entities`, `strategy`, `strategyRationale`, `subTasks[]`

## General Agent Hardening (News Workflows)

The example general agent loop (`agents/general-agent-client.ts`) now includes practical safeguards for news-list goals (for example, "summarize 10 news items"):

- **Completeness gate**: blocks `complete`/`advance_subtask` when collected items are below an 80% threshold of requested count.
- **Smart URL quality filter**: for news goals, extracted items are scored (`high` vs `low` article confidence). If enough high-confidence items exist, low-confidence hub/category URLs are excluded.
- **Deterministic recovery path**: when shortfall is detected, the agent tries infinite scroll re-analysis first, then navigates to fallback listing URLs from learned skills/fixed paths.
- **Prompt smart-chunking for focused content**: prompt context prefers semantic section boundaries over blind character slicing, improving downstream synthesis quality.
- **Post-run scoring**: heuristic evaluator is the default production gate; optional LLM-judge exists for deeper semantic audits.

Important runtime behavior:

- MCP tool timeouts in the general-agent loop are now handled as recoverable tool errors in the shared tool wrapper, so recovery flows do not crash the full session on a single timeout.

## Session Persistence

Persistent profiles are stored under AGENT_EYE_PROFILE_ROOT.

- Cookies are loaded when a persistent profile starts
- Cookies are saved when session manager closes
- Profile name allows separate session histories per workflow/site

### Avoiding Bot Detection & Human Verification

Some websites employ aggressive bot detection (such as CAPTCHAs, Cloudflare walls, or login requirements). To prevent your agent from being blocked:

1. **Warm the session**: Run the warming script to manually open a non-headless browser window:

   ```bash
   npm run warm-session
   ```

2. **Solve challenges manually**: Interact with the page (e.g. solve the CAPTCHA, log in, or accept cookies) in the opened browser window. This saves the verified session cookies to your persistent profile.
3. **Run your agent**: The agent will automatically reuse these saved cookies on subsequent runs to bypass the verification check. Note that cookies and sessions expire over time, so you may need to run `npm run warm-session` again if the agent starts encountering bot blocks.

## Skill Management

Agent Eye features a project-local skill caching system that maps CSS selectors to page structures to avoid redundant LLM page-analysis queries.

### 1. How Skills Are Learned & Saved

- When the agent visits a page and calls the `browser_analyze_page` tool, the system checks if a matching skill pattern already exists.
- If no skill exists, the tool splits the page into chunks, summarizes them using the local LLM, and builds a structural `PageMap` (identifying main content vs noise selectors).
- This `PageMap` is automatically saved as a JSON skill file under the local `./agent-skills/<domain>.json` directory.
- Consecutive page loads on the same domain increments the `successCount` of the skill.
- Each pattern also tracks success/failure metadata so stale selectors can be detected and refreshed automatically.

### 2. Matching and URL Routing

- Skill matching is entirely **URL-driven** (not prompt-driven).
- When a page is analyzed, the system extracts the domain and matches specific path patterns using the first two directory segments (e.g., `vnexpress.net/the-thao`).
- If no exact path-level pattern matches the current URL, the system falls back to the top-level domain pattern (`vnexpress.net`).

### 3. Managing Skills via CLI

You can inspect, refresh, or remove learned domain skills using the Skill CLI:

```bash
# List all learned domain skills
npx tsx agents/skill-cli.ts list

# View the full JSON pattern for a specific domain
npx tsx agents/skill-cli.ts show vnexpress.net

# Delete a skill (forces the agent to re-analyze page structure on next visit)
npx tsx agents/skill-cli.ts delete vnexpress.net
```

Alternatively, you can force the agent to overwrite an existing skill dynamically by calling `browser_analyze_page` with the `"forceReanalyze": true` argument.

### 4. Stale Skill Auto-Refresh

- When a cached pattern returns very low focused content and poor selector matches, the system treats the skill as stale.
- The stale pattern is marked with a failure increment, then `browser_analyze_page` immediately falls back to fresh analysis in the same call.
- For general-agent loops, repeated low-quality `browser_extract_section` results trigger a deterministic escalation:
  1. Force one `browser_analyze_page` refresh
  2. If still failing, escalate to `browser_ocr_chunk`

## Security & Quality

**Code Quality**:

- Pre-commit git hooks (Husky) validate all tests before commit
- Enable with `npm run prepare`
- Hook lives at `.husky/pre-commit` and runs `npm test`

**License**:

- AGPL-3.0: Free for non-commercial and personal use
- Commercial use requires explicit permission

**Extraction Safety**:

- LLM confidence scoring prevents low-reliability extractions
- Field-specific thresholds apply per-field validation
- Evidence tracking for debugging extraction failures
- browser_evaluate_js restricted mode blocks dangerous API patterns unless explicitly unsafe
- Optional domain security blocks navigation to disallowed domains
- Memory guard blocks new navigations when critical threshold is reached

## Tests

### Unit & Integration Tests

```bash
npm run test
```

Relevant test files:

- `tests/ai/goal-planner.test.ts` — GoalPlanner unit tests (12 tests, all mocked, no Ollama required)
- `tests/ai/skill-manager-matching.test.ts` — URL-aware skill matching
- `tests/ai/page-analyzer-dom.test.ts` — DOM candidate selector extraction
- `tests/ai/structured-items.test.ts` — structured item extraction with lazy-load images
- `tests/dom/llm-dom-extractor-phases.test.ts` — 5-phase DOM extraction (16 tests)
- `tests/integration/browser-tools.test.ts` — screenshot SPA hydration
- `tests/integration/huntrix-golden-e2e.test.ts` — YouTube metadata end-to-end
- `tests/integration/ai-orchestrator-live.test.ts` — live Ollama orchestration (opt in: `RUN_LIVE_AI_ORCHESTRATOR_TEST=true`)

### Eval Tests (README 3.1–3.5)

End-to-end behavioral tests that spawn the full `general-agent-client.ts` loop and assert the output matches the README examples. Require Ollama + network access.

```bash
# Run all eval scenarios (CNN, YouTube, OpenAI/Claude, Alphaus, Lynk & Co)
RUN_EVAL_TESTS=true npx vitest run tests/eval/agent-eval.test.ts

# Run a single scenario
RUN_EVAL_TESTS=true npx vitest run tests/eval/agent-eval.test.ts -t "3.1"

# Use a different model
RUN_EVAL_TESTS=true OLLAMA_MODEL=llama3:8b npx vitest run tests/eval/agent-eval.test.ts
```

Covered scenarios:

| Test        | Goal                    | Pass criteria                                                          |
| ----------- | ----------------------- | ---------------------------------------------------------------------- |
| **3.1**     | CNN 10 news             | ≥10 numbered items, contains `cnn.com` URL                             |
| **3.2**     | YouTube Golden metadata | Answer mentions views and/or release date                              |
| **3.3**     | OpenAI vs Claude        | Mentions OpenAI/GPT and Claude/Anthropic                               |
| **3.4**     | Alphaus Cloud info      | Contains company name + a product name                                 |
| **3.5**     | Lynk & Co 08 HCMC price | Contains car name + a VND/price value                                  |
| **Planner** | VnExpress 10 news       | Planner ran, ≥2 sub-tasks, answer >80 chars, no hallucinated selectors |

## Client Integration & Production Simulation

For general browser automation goals, you should drive the browser step-by-step from your own orchestration service (as `ai_orchestrate_workflow` is optimized specifically for YouTube metadata extraction).

### 1. General Agent Loop Client Script

We provide a production-ready simulation client script in [agents/general-agent-client.ts](agents/general-agent-client.ts) that demonstrates how to:

- Connect to the `agent-eyes` MCP server over stdio transport.
- Run the **GoalPlanner preamble** once before the step loop: sanitize the goal, select a strategy, and emit ordered sub-tasks with step budgets.
- Query a local Ollama model (e.g. `gemma4:e4b-it-q4_K_M`) per step — each step prompt shows only the _active sub-task_ instead of the full raw goal, keeping the context small.
- Advance sub-tasks via `advance_subtask: true` in the action JSON; re-plan automatically when a sub-task exceeds its budget.
- Run actions step-by-step using MCP tools.

### 2. Running Simulations

You can evaluate the system on any prompt goal using:

```bash
# Run with default goal (VNExpress news summary)
npx tsx agents/general-agent-client.ts

# Run with a custom prompt
npx tsx agents/general-agent-client.ts "search what is the best trend in github.com"
```

### 3. Capturing Debug Screenshots

For easier debugging, you can enable step-by-step browser screenshots by appending the `--screenshot` CLI option. This will capture viewports at each step using the `browser_screenshot` tool and write them to a local `debug-screenshots/` directory:

```bash
npx tsx agents/general-agent-client.ts "search what is the current stock for VCB (HOSE vn)" --screenshot
```

### 3.1 Example: CNN News Summary (10 items)

Example command:

```bash
npx tsx agents/general-agent-client.ts "go to cnn.com and summary for me list 10 of news" --screenshot
```

Expected behavior:

- Agent navigates to `https://edition.cnn.com/`
- Runs `browser_analyze_page` and extracts structured `items[]` with source URLs/images
- Returns exactly 10 news summaries with source links

```text
🎉 Goal Reached!

Based on the visible content of CNN.com from June 28, 2026, here are 10 news summaries visible on CNN.com:

1. Germany's new Nazi party databases are challenging decades-held sanitized family narratives
  Source: https://edition.cnn.com/world/germans-nazi-past-far-right-intl

2. Israel's military and tech industry race to counter Hezbollah's latest threat
  Source: https://edition.cnn.com/world/middleeast/israel-tech-hezbollah-drone-threat-intl

3. The US and Iran have a deal on paper. At sea, the Strait of Hormuz is 'chaotic'
  Source: https://edition.cnn.com/world/iran-us-hormuz-agreement-mou-intl

4. Live updates: Death toll climbs to over 1,400 in Venezuela quakes
  Source: https://edition.cnn.com/world/live-news/venezuela-earthquake-hnk
```

### 3.2 Example: YouTube Video Metadata (likes, views, release date)

Example command:

```bash
npx tsx agents/general-agent-client.ts "search Golden Official, to see that video has how many like, view and release date" --screenshot
```

Expected behavior:

- Agent runs YouTube-focused search via `browser_search` with `site="youtube"`
- Navigates to the selected watch URL
- Uses `browser_analyze_page` on the watch page to extract metadata
- Returns video title, views, release date, and likes (if visible)

Sample output excerpt:

```text
🎉 Goal Reached!

Based on the provided YouTube page content for "Golden" Official Lyric Video | KPop Demon Hunters | Sony Animation:

- Video Title: “Golden” Official Lyric Video | KPop Demon Hunters | Sony Animation
- Views: 1,531,178,811 views
- Release Date: Jun 23, 2025
- Likes: The explicit number of likes is not visible in the provided focused content.
```

### 3.3 Example: OpenAI vs Claude News + Benchmark Table

Example command:

```bash
npx tsx agents/general-agent-client.ts "list me 5 news about AI models from OpenAI and Claude, which one is better, I need some benkmark or even the table for easy compare features" --screenshot
```

Expected behavior:

- Agent starts from Google search and gathers relevant comparison sources
- Navigates into a detailed article and runs `browser_analyze_page`
- Returns a structured comparison with benchmark values and an easy-to-read table
- Includes a practical recommendation on “which one is better” by use case

Sample output excerpt:

```text
🎉 Goal Reached!

Comparative Overview (Benchmarks Table)

| Category | GPT-5.4 (OpenAI) | Claude Opus 4.6 (Anthropic) | Gemini 3.1 Pro | Grok 4 |
| :--- | :--- | :--- | :--- | :--- |
| Coding | Strong (74.9% SWE-bench) | Strong (74%+; powers Cursor) | Good (63.8%, 1M context) | Leader (75%) |
| Reasoning | 92.8% GPQA | 91.3% GPQA | Leader (94.3% GPQA) | Competitive |
| Writing | Good (Canvas editor) | Leader (128K output, natural prose) | Good (Docs integration) | Uncensored style |
| API Price (in/out per 1M) | $2.50/$15 | $15/$75 (Opus), $3/$15 (Sonnet) | $2/$12 | $2/$15 |

Summary on which is better:
- Claude Opus 4.6 is stronger for long-form writing and natural prose.
- OpenAI GPT-5.4 remains a strong all-rounder with broad ecosystem support.
```

### 3.4 Example: Company Information Discovery (Alphaus Cloud)

Example command:

```bash
npx tsx agents/general-agent-client.ts "help me to find information about Alphaus Cloud company" --screenshot
```

Expected behavior:

- Agent starts with `browser_search` on Google to gather initial company sources
- Navigates to the official website (`https://www.alphaus.cloud/en`)
- Runs `browser_analyze_page` and refreshes stale skill selectors when needed
- Returns a structured company profile: mission, credentials, products, and services

Sample output excerpt:

```text
🎉 Goal Reached!

Based on the Alphaus Cloud landing page, here is detailed information about the company:

Company overview and mission:
- Alphaus Cloud focuses on cloud cost intelligence and FinOps-driven optimization.

Key credentials:
- Claims No. 1 position in Japan with over $200M+ cloud resources managed annually.
- Preferred FinOps solution by 25% of AWS Premier Partners in Japan.
- Reports 3000+ monthly active users on Ripple.

Core products and services:
- Octo: cost visibility and optimization for cloud users.
- Ripple: billing automation and reseller-focused cloud cost management.
- Professional Services: FinOps implementation and cross-team enablement.
```

### 3.5 Example: Localized Product + Price Discovery (Lynk & Co 08 in HCMC)

Example command:

```bash
npx tsx agents/general-agent-client.ts "find information about Lynk & co 08 in Vietnam? and current price for that car in Ho Chi Minh?" --screenshot
```

Expected behavior:

- Agent runs localized search with Vietnamese keywords via `browser_search`
- Uses focused content to extract product profile and local on-road pricing
- Returns model details and city-specific price estimates for Ho Chi Minh City

Sample output excerpt:

```text
🎉 Goal Reached!

General information on Lynk & Co 08:
- D-segment SUV, PHEV (1.5 Turbo + electric motor)
- EV range up to 200 km, total range over 1,400 km

Current price in Ho Chi Minh City (HCMC):
- Lynk & Co 08 Pro:
  - Factory list price: 1,299,000,000 VND
  - Estimated on-road price (HCMC): 1.43-1.45 billion VND
- Lynk & Co 08 Halo:
  - Factory list price: 1,389,000,000 VND
  - Estimated on-road price (HCMC): 1.53-1.55 billion VND
```

### 4. Integration Code Pattern

To call this MCP server from your own Node.js service, use the following code pattern:

```typescript
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

const transport = new StdioClientTransport({
  command: 'npx',
  args: ['tsx', 'src/index.ts'],
  cwd: '/path/to/agent-eyes',
});

const client = new Client({ name: 'my-orchestrator', version: '1.0.0' }, { capabilities: {} });

await client.connect(transport);

// 1. Navigate
await client.callTool({
  name: 'browser_navigate',
  arguments: { url: 'https://vnexpress.net' },
});

// 2. Get simplified page content for LLM context
const contentResult = await client.callTool({
  name: 'browser_get_content',
  arguments: { format: 'simplified-html', includeInteractiveMap: true },
});
const textContent = (contentResult as any).content?.find((c: any) => c.type === 'text')?.text;
const payload = JSON.parse(textContent);
console.log(payload.content); // Simplified HTML content
console.log(payload.interactiveMap); // Actionable ID mapping

// 3. Interact (e.g. click element with data-agent-id="34")
await client.callTool({
  name: 'browser_interact',
  arguments: { action: 'click', target: '34' },
});

// 4. Capture screenshot for debugging
const screenshotResult = await client.callTool({
  name: 'browser_screenshot',
  arguments: { fullPage: false },
});
const imageItem = (screenshotResult as any).content?.find((c: any) => c.type === 'image');
if (imageItem?.data) {
  // imageItem.data contains raw base64 PNG data
  fs.writeFileSync('debug-step.png', Buffer.from(imageItem.data, 'base64'));
}

await client.close();
```