The OmniBrowse-Agent is a controlled browser automation server that enables AI agents to navigate, interact with, and extract data from public websites. It offers tools for browser navigation, clicking marked elements, scrolling, session and tab management, taking snapshots, performing general actions, and extracting content in various modes (e.g., text, links, tables, ARIA). As an evidence-centric web research agent, it generates structured reports with source citations and confidence levels, enforces budget limits on steps and costs, and requires human intervention for sensitive operations. It operates within strict security boundaries, blocking private network access and validating page versions to prevent stale interactions, while providing real-time observability through a web demo.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@OmniBrowse-Agent Extract the key facts from this webpage and save a screenshot as evidence."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
OmniBrowse-Agent 是一个以证据为核心的浏览器研究 Agent：接收研究问题后，使用真实浏览器（Playwright）在公开网页上导航、点击、滚动、输入和采集，优先通过 DOM、ARIA、表格及网络响应提取事实，必要时再调用视觉模型兜底，最后生成带来源、采集时间、置信度和证据引用的 Markdown 报告。成本、步数和视觉调用受预算闸门控制；登录、验证码及敏感操作转人工处理，不绕过站点安全机制。
当前定位：可演示、可测试、可扩展的单机工程原型。未实现能力会在 Roadmap 中明确标注。
启动本地 Web Demo 后，在浏览器中输入研究问题即可看到实时浏览画面、Set-of-Marks（SoM）标记、动作轨迹、事实与证据，以及最终报告。
演示流程支持两种入口：问题中包含 http(s):// 绝对网址时直接导航；未包含网址时从默认百度搜索页开始。遇到登录或验证码页面，任务会暂停，用户在可见浏览器中完成处理后点击恢复，系统会重新检查页面状态。
Related MCP server: @meshbrow/mcp-server
Web 演示：python run_web.py 启动本地界面，默认监听 http://127.0.0.1:8766/；展示浏览帧、事件流、AI 分析、事实证据、成本和报告。
受控浏览器：Playwright Chromium、独立 BrowserContext、标签页管理、SoM 元素定位、page_version 防止使用过期元素映射。
浏览动作：导航、点击、输入、滚动、后退和选择；敏感或状态变更动作需要人工确认。
确定性提取：文本、标题、链接、ARIA、表格、JSON-LD、SVG 文本、网络响应和 ECharts 数据。
长页与晚渲染处理：模型侧页面文本上限 24,000 字符；对可滚动长页按需滚动采集；交互元素不足时进行有界等待；无效果动作和无进展指纹用于终止空转。
模型路由：普通文本决策使用配置的 cheap_text 模型（默认 GLM，支持本地 Ollama 兜底）；视觉只在需要时使用，支持 GLM 视觉接口或本地 qwen2.5-vl。
证据化结果：SQLite 保存任务、步骤、事实、证据、模型调用、预算事件和报告；报告包含来源 URL、页面版本、提取方法、置信度和证据片段。
预算控制：使用整数 microcny 账本（1 CNY = 1,000,000 microcny），可限制成本、Token、步数、页面数、时长、视觉调用和强推理调用。
CLI 与 Browser MCP：提供 omnibrowse run/resume 任务命令，以及 stdio 传输的 omnibrowse-browser-mcp 服务。
安全边界：仅允许 HTTP(S)，出口代理在 DNS/IP 层阻止回环、私网、链路本地和云元数据地址；网页内容作为不可信数据，不能修改安全策略或预算。
测试与 CI：pytest 分层测试（unit / contract / integration / e2e）与 GitHub Actions 的 Python 3.11/3.12 检查。
py -3.11 -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
python -m playwright install chromium
Copy-Item .env.example .env
python run_web.py然后打开 http://127.0.0.1:8766/。如需填写 GLM API Key，请只写入本地、不会提交到 Git 的 .env 文件；未配置外部模型时仍可运行本地流程，但 AI 分析能力会受配置影响。
也可以使用已安装的入口：
omnibrowse-web --port 8765TaskSpec 使用 JSON 文件描述目标网页、提取方法和事实规则：
omnibrowse run --task-spec task.json
omnibrowse resume --task-id <task_id>命令输出结构化 JSON，报告和制品路径由配置中的数据库与 artifacts 目录决定。
omnibrowse-browser-mcp服务按需启动 Chromium。工具层会校验 URL、会话、标签页、page_version、动作类型和提取模式。
复制 .env.example 为 .env 后按需修改。配置按用途分为以下前缀：
前缀 | 用途 |
| 运行环境、日志、SQLite 数据库和 artifacts 路径 |
| headless、视口和浏览器超时 |
| 文本、视觉模型及其端点 |
| 成本、Token、步数、页面、时长和视觉调用上限 |
| 制品保留时间与清理周期 |
密钥、Cookie、Token 和真实账号数据不得提交到仓库。
flowchart TD
U[用户研究问题] --> W[Web Demo / CLI]
W --> G[有界工作流]
G --> B[Browser Gateway]
B --> M[Browser MCP]
M --> P[Playwright Chromium]
B --> X[确定性提取器]
X --> F[SQLite 事实与证据账本]
X --> Q{确定性证据足够?}
Q -->|否且预算允许| V[视觉模型兜底]
V --> F
Q -->|是| R[报告渲染器]
F --> R
R --> O[Markdown 报告]src/omnibrowse_agent/web/：本地 Web Demo、任务状态、事件、人工恢复和展示接口。
src/omnibrowse_agent/graph/：任务状态流转、浏览/提取/验证/写作和预算终止。
src/omnibrowse_agent/browser_gateway/：对 Browser MCP 的结构化网关与 stdio 客户端。
src/omnibrowse_agent/models/：文本动作决策、答案合成和视觉模型适配器。
src/omnibrowse_agent/extraction/：声明式、确定性事实提取。
src/omnibrowse_agent/storage/：SQLite 迁移、任务/事实/证据/报告持久化和制品登记。
browser_vision_mcp/：Playwright、SoM、提取、artifact 和受控网络出口实现。
OmniBrowse-Agent/
├── src/omnibrowse_agent/ # 应用、编排、模型、存储与报告
├── browser_vision_mcp/ # Browser MCP 服务及浏览器安全策略
├── tests/ # unit / contract / integration / e2e
├── docs/ # 设计书、ADR、术语表与演示截图
├── .env.example # 配置模板
└── run_web.py # 8766 端口 Web Demo 启动脚本ruff check .
pytestGitHub Actions 会在 Python 3.11 和 3.12 上安装开发依赖并执行 Ruff 与 pytest。更完整的架构、契约和安全说明见 项目设计书、交接文档 和 AGENTS.md。
仅访问用户授权范围内的公开 HTTP(S) 页面；不访问 file://、内网或云元数据地址。
CAPTCHA、登录、双因素认证、上传、提交、支付、删除、发布及其他敏感操作必须人工处理或确认。
网页中的文字和指令是不可信数据，不能覆盖系统目标、预算、安全策略或工具权限。
DOM、ARIA、表格和网络数据优先于模型；视觉模型只作为局部兜底，且受调用次数和成本上限约束。
无法确认的事实保留为 unknown，不通过猜测补全数值。
以下项目属于后续规划，不应视为当前版本承诺：
多租户 SaaS、分布式浏览器集群和高可用会话池。
PDF、下载文件和大体积附件的完整解析流水线。
以向量数据库作为补充的跨页面语义检索（SQLite 仍是事实真源）。
更完整的图表适配器、评测平台和成本回归基准。
OpenTelemetry/Langfuse 等增强型可观测性与远程部署。
本项目以 MIT License 发布。
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes | ||
| tab_id | No | ||
| mark_id | No | ||
| direction | No | down | |
| session_id | Yes | ||
| page_version | No |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Tool has no description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Tool has no description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has no description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Tool has no description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Tool has no description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Tool has no description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Click an element from the latest Set-of-Marks snapshot.
| Name | Required | Description | Default |
|---|---|---|---|
| mark_id | Yes |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full responsibility for behavioral disclosure. It only states 'Click an element' without addressing side effects like navigation, failure behavior if no snapshot exists, or what the response contains. The snapshot dependency is noted, but other behavioral traits remain opaque.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no unnecessary words. It is front-loaded with the action and target, making it easy to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter tool, the description still lacks essential context: what Set-of-Marks means, where marks come from, what prerequisites must hold, and what outcome to expect. The existence of an output schema does not compensate for the absence of these operational details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage and the description never mentions mark_id, how to obtain it, or its relationship to the Set-of-Marks snapshot. The description fails to add any meaning beyond the bare schema definition of an integer parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Click') and resource ('element from latest Set-of-Marks snapshot'), clearly distinguishing it from navigation, scrolling, and generic action tools. The mention of 'Set-of-Marks' adds specificity beyond what the name alone implies.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'from the latest Set-of-Marks snapshot' implies a clear prerequisite (a snapshot must exist) and contextual usage, but it does not explicitly name alternatives like browser_act or state when not to use this tool. This provides clear context without exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| tab_id | No | ||
| element_id | No | ||
| session_id | Yes |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Tool has no description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Tool has no description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has no description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Tool has no description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Tool has no description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Tool has no description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scroll the active viewport up or down and return a fresh snapshot.
| Name | Required | Description | Default |
|---|---|---|---|
| direction | No | down |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses the core behavior (scroll, snapshot) but lacks detail on edge cases (e.g., boundary behavior, waiting for content) or potential side effects. It does not contradict annotations (none exist), but the disclosure is minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is a single, well-structured sentence that front-loads the action and outcome. Every word earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple (one optional parameter, output schema provided), and the description adequately covers the purpose and behavior for basic usage. It could mention edge cases or prerequisites, but given the output schema and simplicity, it is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has a single 'direction' parameter with no description (0% coverage). The description compensates by specifying 'up or down', giving the parameter meaningful allowed values. This is helpful but not exhaustive (e.g., no mention of default behavior).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action ('Scroll') and the target resource ('active viewport'), and explicitly mentions the result ('return a fresh snapshot'). It distinguishes itself from sibling tools by being the only scrolling-related tool, with clear specificity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage is clear: use when you need to scroll the viewport and get a snapshot. However, it does not provide explicit when-not-to-use guidance or alternative tool references, which prevents a score of 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
| Name | Required | Description | Default |
|---|---|---|---|
| session_id | Yes | ||
| cleanup_artifacts | No |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Tool has no description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Tool has no description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has no description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Tool has no description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Tool has no description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Tool has no description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Tool has no description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Tool has no description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has no description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Tool has no description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Tool has no description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Tool has no description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
| Name | Required | Description | Default |
|---|---|---|---|
| tab_id | No | ||
| session_id | Yes |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Tool has no description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Tool has no description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has no description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Tool has no description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Tool has no description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Tool has no description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
| Name | Required | Description | Default |
|---|---|---|---|
| operation | No | list | |
| session_id | Yes |
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| data | No | |
| error | No | |
| usage | Yes | |
| tab_id | No | |
| trace_id | Yes | |
| artifacts | No | |
| session_id | No | |
| page_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Tool has no description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Tool has no description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has no description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Tool has no description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Tool has no description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Tool has no description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0browser_actbrowser_click_markbrowser_extractbrowser_navigatebrowser_scrollbrowser_session_closebrowser_session_createbrowser_snapshotbrowser_tabsScored across 9 tools
Several tools have unclear or missing descriptions. browser_act, browser_snapshot, and browser_extract overlap in purpose without clear boundaries. The generic 'act' could subsume click_mark or scroll, and 'snapshot' may duplicate the marked viewport from navigate.
All tools share the 'browser_' prefix, which is good, but the second part mixes verb phrases (navigate, click_mark, scroll) with noun phrases (tabs, snapshot). 'browser_act' is too vague and breaks the more descriptive pattern.
9 tools is a reasonable size for a browser automation server. The count is not excessive and covers major browsing actions, though some tools could be consolidated.
The server covers core browsing actions like navigation, clicking, scrolling, session management, and tabs. However, there are potential gaps such as form fill, back/forward, or wait operations, and the generic 'act' and 'extract' tools leave it unclear whether those capabilities are fully addressed.
A paid remote MCP for AI agent browser approval MCP, built to return verdicts, receipts, usage logs,
A paid remote MCP for AI agent browser MCP session, built to return verdicts, receipts, usage logs,
Stealth web browser for agents: search, fetch, click, download and type in persistent MCP sessions.
Hosted MCP catalog with 30 tenant-isolated browser, RAG, AI, mail and media tools.
