Autonomous Browser Agent — no screenshots

The agent drives a real Chromium via Playwright, and the LLM makes decisions based on a structured textual snapshot of the page (ref | role | name | states), not images. No vision model is needed, and no screenshots are taken at any step.

Three operating modes from a single codebase:

Architecture

Related MCP server: Playwright MCP Server

1. Installation on a local machine

git clone https://github.com/maxmkab/autonomous-browser-agent.git
cd autonomous-browser-agent

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\\Scripts\\activate
pip install -r requirements.txt
playwright install chromium

cp .env.example .env             # вписать ANTHROPIC_API_KEY

Check without spending on the LLM — see exactly what the model sees:

python main.py snapshot --url https://example.com

Run a task with a visible browser:

HEADLESS=false python main.py run \
  --task "Найди раздел с ценами и извлеки все тарифы через extract" \
  --url https://example.com \
  --json state/report.json

2. Embedding into your real browser (CDP)

The agent can work not in its own clean Chromium but in your Chrome — with live sessions, extensions, and a real fingerprint. No extensions need to be installed — control happens over the Chrome DevTools Protocol.

# 1) запустить Chrome с открытым портом (отдельный профиль для агента)
chmod +x scripts/chrome-cdp.sh
./scripts/chrome-cdp.sh 9222          # Windows: scripts\\chrome-cdp.bat 9222

# 2) в другом терминале отдать задачу агенту в этом же браузере
CDP_URL=http://127.0.0.1:9222 python main.py run --task "..."

In this mode the agent does not touch storage_state.json: sessions are taken from the Chrome profile. On first launch, log into the services you need by hand; the profile remembers them from then on.

3. Embedding into Claude Code / Cursor via MCP

mcp_server.py starts an MCP server with stdio transport. Tools: browser_open, browser_snapshot, browser_click, browser_type, browser_select, browser_check, browser_scroll, browser_press, browser_back, browser_tabs, browser_save_session, browser_run_task, browser_close.

Config for Claude Code (~/.claude.json or .mcp.json in the project root):

{
  "mcpServers": {
    "browser-agent": {
      "command": "/absolute/path/autonomous-browser-agent/.venv/bin/python",
      "args": ["/absolute/path/autonomous-browser-agent/mcp_server.py"],
      "env": {
        "HEADLESS": "false",
        "CDP_URL": "http://127.0.0.1:9222",
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "REQUIRE_APPROVAL": "true"
      }
    }
  }
}

Or with a single command:

claude mcp add browser-agent -- /absolute/path/.venv/bin/python /absolute/path/mcp_server.py

After that, the model in Claude Code drives the browser in a loop browser_snapshot → browser_click → browser_snapshot, receiving only textual snapshots. The browser session lives between calls, so a scenario can be driven step by step. For full autonomy there is browser_run_task — the agent runs the loop itself and returns a JSON report.

4. Moving authorization to the server

# локально, в видимом окне: залогинился → Enter в консоли
HEADLESS=false python main.py login --url https://site.ru/login

# переносим cookies + localStorage на сервер
scp state/storage_state.json root@YOUR_VPS_IP:/opt/browser-agent/state/

5. Deploying to a VPS (Ubuntu 24.04 + Docker)

mkdir -p /opt/browser-agent/state && cd /opt/browser-agent
git clone https://github.com/maxmkab/autonomous-browser-agent.git .
cp .env.example .env && nano .env

docker build -t browser-agent .
docker run -d --name browser-agent --restart unless-stopped \
  --shm-size=1g \
  --env-file .env \
  -v /opt/browser-agent/state:/app/state \
  browser-agent

--shm-size=1g is mandatory: Chromium in a container with the default 64 MB /dev/shm crashes on heavy pages.

One-off task on the server:

docker exec -it browser-agent python main.py run --task "..." --url https://...

Without Docker (systemd)

apt update && apt install -y python3-venv
cd /opt/browser-agent && python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
playwright install --with-deps chromium

/etc/systemd/system/browser-agent.service:

[Unit]
Description=Autonomous browser agent
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/opt/browser-agent
EnvironmentFile=/opt/browser-agent/.env
ExecStart=/opt/browser-agent/.venv/bin/python main.py daemon --interval 300
Restart=always
RestartSec=10
StandardOutput=append:/var/log/browser-agent.log
StandardError=append:/var/log/browser-agent.log

[Install]
WantedBy=multi-user.target

systemctl daemon-reload && systemctl enable --now browser-agent
journalctl -u browser-agent -f

6. Task submission and n8n

The daemon reads state/tasks.jsonl — one line = one task:

{"id":"price-check-1","task":"Открой карточку товара, извлеки цену и наличие через extract","url":"https://site.ru/item/123"}
{"id":"lead-form","task":"Заполни форму заявки: имя Иван, телефон +79990000000. Отправку подтвердит человек."}

Results are written to state/results.jsonl with the fields success, result, extracted, steps, tokens_in/out, and the full history trace. n8n can write tasks to this file (Execute Command / SSH node) and read the results.

7. Token savings

• No screenshots: text only, no vision model.

• DOM filtering: only visible interactive elements with a non-empty name go into the context, 250 max per frame.

• Image/font/media blocking at the network level (BLOCK_MEDIA=true).

• History compression (HISTORY_WINDOW): full snapshots only for the latest steps, older ones are collapsed into action → result.

• Loop detection: if the snapshot fingerprint does not change, the model is instructed to change strategy.

8. Security

Actions matching RISKY_PATTERNS (payment, buy, order, delete, send, checkout, pay, delete) require approval: in the console locally or a "yes" reply in Telegram on the server (APPROVAL_MODE=telegram). ALLOW_EVAL=false by default forbids executing arbitrary JS. All action errors are returned to the model as observations and do not crash the process. Secrets are stored only in .env, which is excluded from git.

9. Pre-production checks

1. python main.py snapshot --url <target site> — do the needed elements appear in the snapshot?

2. Run the task locally in headful with REQUIRE_APPROVAL=true.

3. The same task locally in headless — catches rendering differences before deploy.

4. Only then deploy to the VPS and start the daemon.

License

MIT