Bosun

The boatswain browser for agents. A self-hosted browser service that gives AI agents a real browser — and calls a human on deck when it needs one (CAPTCHA, cookie wall, 2FA).

Status: prototype (M0). Works end to end in one deployment; APIs will change. See docs/DESIGN.md and CHANGELOG.md.

What it does today

• Headful Chromium (Playwright) under Xvfb — fewer bot flags than headless.

• Engines (ENGINE, or engine on session_open): chromium (default), patchright (Chromium without the CDP/automation leaks) and camoufox (anti-detect Firefox). See docs/ENGINES.md for what each one passes and what it costs.

• MCP server over stdio (Claude Code, Codex, …) or WebSocket (shared service).

• Tools: session_open / session_close / sessions, navigate, read (accessibility snapshot with element refs), click, type, select, press, back, wait_for, screenshot, request_human, handoff_wait, keepalive_set / keepalive_list / keepalive_remove, meeting_join / meeting_status / meeting_chat_read / meeting_chat_send / meeting_stop_recording / meeting_leave / meeting_recording_delete.

• Keepalive for persistent profiles: scheduled visits keep sliding login sessions alive without a model, and a logged-out profile raises an MCPL bosun:needs-human push event. A visit can also watch one value (say, a "43 neue Nachrichten" counter) and push bosun:watch-changed when it goes up. See Keeping logins alive and Watching a value.

• Meetings (PoC): joins a Google Meet as a guest notetaker, announces itself in the chat, records the audio to a file and leaves; a "please don't record" in the chat stops and deletes the recording. Transcription happens later, from the file. See Meetings (PoC).

• Sessions are ephemeral (wiped on close) or use a named persistent profile of the tenant (stays logged in; no RAM while idle). A session belongs to the connection that opened it: other connections can't see, use or close it, and it is closed when that connection goes away or after IDLE_TIMEOUT_SEC without use.

• navigate and keepalive_set accept only http:// and https:// URLs (no file:, data:, javascript:, about:, chrome:, view-source:, blob:, ftp: …).

• Bot-wall detection after navigation/click (Cloudflare challenge pages, CAPTCHA iframes, "access denied" texts). A bot check that keeps reloading itself (IP reputation) is reported as such: bot check keeps reloading (likely IP reputation); try engine chromium + request_human, or another network.

• Human handoff: request_human returns a link (15 min) to a live view of the tab, made for a phone: Trackpad mode (cursor, drag to move, tap to click, two fingers scroll, hold and drag for sliders) or Tap mode, a masked paste bar for passwords and codes, a key strip, a desktop mouse works as usual. The link can be reopened (one viewer at a time) until Done resumes the agent. See Handoff for what is and isn't protected.

Related MCP server: AIHawk

Run

npm ci                                   # add --omit=optional to skip patchright/camoufox
npx playwright install --with-deps chromium
npx patchright install chromium          # only for ENGINE=patchright
npx camoufox-js fetch                    # only for ENGINE=camoufox (Node 22+)
npm test            # headless, against a local test server; no internet needed
npm run test:engines  # the same suite once per installed engine

# stdio (an MCP client spawns it)
xvfb-run -a node src/server.mjs

# WebSocket service + live view, reachable over a VPN interface (here 100.64.0.10)
WS_HOST=100.64.0.10 WS_PORT=7360 BROWSER_TOKEN=<32+ random chars> \
LIVE_HOST=100.64.0.10 LIVE_PORT=7350 LIVE_BASE_URL=http://100.64.0.10:7350 \
  xvfb-run -a -s "-screen 0 1366x900x24" node src/server.mjs

Clients authenticate with Authorization: Bearer <token> (the old ?token= query parameter still works but puts the secret in URLs and logs; prefer the header). The WebSocket listener refuses to start on a non-loopback address without a token; on loopback an unauthenticated listener needs ALLOW_NO_TOKEN=1 (local development only). The two listeners are separate: WS_HOST/WS_PORT carry MCP (token-protected), LIVE_HOST/LIVE_PORT serve the human's live view (protected by the handoff link). Both default to 127.0.0.1; bind them to a VPN/mesh address or put a TLS reverse proxy in front, and set LIVE_BASE_URL to the address the human's phone uses.

Environment

All numbers are validated at startup; a bad value stops the service instead of silently disabling a limit.

Engines

A session asking for an engine that isn't installed gets an error naming the install command; there is no silent fallback to another engine. A persistent profile remembers its engine family (Chromium or Firefox) and refuses the other. patchright and camoufox-js are optionalDependencies; their browsers are separate downloads.

Chromium spawns many threads: under systemd give the unit a generous TasksMax (e.g. 2000).

Handoff

• One viewer at a time, reopenable until the end. While the handoff runs, the link opens the live view whenever nobody else is connected — closing a chat app's in-app browser and tapping the link again works. A second viewer is refused (its page says the link is open on another device and retries) until the first disconnects or has been silent for more than 5 s. Opening the page alone (e.g. a link preview) doesn't connect. Done, Give up, expiry, session_close and disconnect of the agent end the handoff; from then on the link is dead (404).

• Controls (bottom bar): Tap | Trackpad (trackpad is the default on touch screens), Keys (Tab, Enter, ⌫, Esc, arrows, and the phone keyboard), the paste bar, Give up, Done. Trackpad: one finger moves the cursor (accelerated), tap clicks at the cursor, two fingers scroll, hold then drag holds the button down. The service moves the page's mouse along an interpolated, slightly jittered path rather than jumping.

• Paste bar: paste (long-press → Paste on iOS) or type into the masked field, then Insert or Enter: the text goes into the page's focused field (tap that field first) and the bar is cleared. The eye button shows what's in it. The text is not logged, not kept by the service after it is typed, and never sent to the agent — but once it is in the page, the page field rules below apply (a password field is masked; an ordinary field is not).

• A reconnecting viewer gets the last frame at once, and a page that doesn't repaint is re-captured every 2 s.

• While a handoff is active every page tool (read, screenshot, navigate, click, type, select, press, back, wait_for) is refused, and sessions/handoff_wait report no URL or title. Tool calls that were already running when request_human came in finish first; the human gets the tab after them.

• handoff_wait waits 50 s by default (below the MCP SDK's 60 s request timeout); call it again on still_waiting. Longer timeoutSec needs a longer client-side request timeout.

• After the handoff the agent can read the page again. What the human typed into password, PIN, OTP and card security-code fields (detected by input type, autocomplete, name/id and label, across frames and open shadow roots, including show-password toggles) is masked as [masked] in read output and blacked out in screenshots. After Give up / timeout / cancel those fields are emptied before the agent resumes; if that fails, the session is closed instead. If a frame can't be inspected, read/screenshot fail rather than return unmasked output. The handoff ends only after the human's last input has reached the page.

• A password field that is revealed with a show-password toggle stays masked: an event-driven tracker remembers every input that ever was type=password (incl. shadow roots), and Bosun also notes them when a handoff starts and every 500 ms during it. On chromium Bosun's helpers run in the page's own JavaScript world, so a page can detect them; on patchright and camoufox they run in an isolated world (details in docs/ENGINES.md).

• Not protected: text the human types into ordinary fields, secrets the page itself echoes as normal text, custom widgets that aren't <input>/<textarea> (e.g. per-digit OTP boxes built from divs), closed shadow roots. Those are visible to the agent after Done. Ask the human to finish the login (submit it) before pressing Done.

Keeping logins alive

A persistent profile stays logged in only as long as the site's session does. Many sites extend the session each time it is used ("sliding" sessions) and end it after some idle time. Bosun can visit a profile on a schedule so that idle time never runs out — without a model:

// once, after the human logged the profile in (session_open({profile:"shop"}) + request_human)
keepalive_set({ "profile": "shop", "url": "https://shop.example/account",
                "every_hours": 12, "jitter_min": 30,                     // defaults; every_hours >= 1
                "logged_in_check": { "text": "My account" } })          // or selector / url_not_contains
keepalive_list()     // schedule, next run, last results: ok | logged_out | bot_check | error
keepalive_remove({ "profile": "shop" })

The interval is every_hours (default 12) or every_minutes (at least 30), each visit shifted by up to ± jitter_min (default 30, or a quarter of a shorter interval). quiet_hours ("22-08", "22:30-07:00", local time in TZ, default Europe/Vienna) makes the interval four times longer while it lasts, so a 30-minute watch visits about every two hours at night. An interval that starts before a quiet window is stretched only for its part inside it. Quiet hours follow the wall clock, and on DST nights the time is counted as it really passes. Calling keepalive_set again replaces the whole configuration of that profile.

Each visit, at the interval ± jitter_min (first one an interval after keepalive_set):

• opens the profile with the engine and mode it was last used with (recorded in the profile; a profile used headful is never opened headless, so the fingerprint stays the same);

• loads url, then for 20–60 s moves the mouse along human-looking paths (the live view's trackpad interpolation), scrolls 1–3 times and sometimes rests over a link. It never clicks, types or navigates;

• checks logged_in_check (every given condition must hold: text visible, selector visible, URL doesn't contain url_not_contains), reads the watched value if there is one, closes the profile and records the result (time, status, duration; last 10 kept).

Visits run one at a time, a few minutes apart, and never take the last free session slot from agents. A profile an agent has open is skipped (it's in use anyway); an agent opening a profile during a visit takes it over at once. Schedules and results live in BROWSER_DATA/keepalive.json (mode 0600) and survive restarts. Nothing from the page is stored or logged beyond the check result and the watched value.

When a visit finds the profile logged out or stuck at a bot check, Bosun pushes an MCPL event to the tenant's agent (feature set bosun.handoff, tags bosun:needs-human, bosun:source=keepalive, bosun:reason=logged-out|bot-check, bosun:profile=<name>; origin {profile, url, reason}), so the agent can open the profile and call request_human. Events are only sent to connections whose host asked for MCPL at initialize and granted pushEvents (featureSets/update); if none is connected, up to 20 per tenant wait for the next one. There is no retry loop: the next attempt is the next scheduled visit.

Limits — what this can't do:

• It only helps sites with sliding sessions. A cookie with an absolute expiry (e.g. "remember me for 30 days") ends on that date however often the profile is visited.

• Device re-verification, periodic re-login and 2FA prompts still need a human (that's what the event is for).

• Visits look like a person glancing at the page, but too-frequent visits can look bot-like, especially at the same time every day — hence the 12 h default and the jitter. Don't go below a few hours unless the site's idle timeout requires it.

• The visit runs from Bosun's IP with Bosun's browser; if the site ties the session to a different network or device, keeping it alive here doesn't help.

Watching a value

A visit can read one value from the page after the logged-in check and tell the agent when it changes. For willhaben's message counter in the header:

keepalive_set({ "profile": "willhaben", "url": "https://www.willhaben.at/iad/myprofile/chat",
                "every_minutes": 30, "quiet_hours": "22-08",
                "logged_in_check": { "url_not_contains": "/login" },
                "watch": { "text_regex": "(\\d+) neue Nachrichten", "kind": "count", "label": "willhaben messages" } })

• What is read: the selector's textContent if given, else the page's visible text. text_regex picks the value (its first capture group, else the whole match). The regex runs in the page with a 5 s limit, so a pathological pattern can't stall the service.

  • kind: "count": the first whole number in the text. Thousands separators inside it are ignored, but the regex has to capture them: use ([\d.]+) rather than (\d+) if the site writes "1.234". No match counts as 0, since many badges disappear when nothing is new.

  • kind: "text": the text with whitespace collapsed, at most 200 characters. No match is null.

• When it is reported: the first successful read is the baseline and raises no event. After that, a count that went up or a text that differs raises one bosun:watch-changed event. A count that went down (you read the messages) is stored silently, and the next rise is measured from there.

  • Nothing is read when the visit isn't logged in.

  • A read that fails is recorded as watch timeout, no number in the watched text or watch failed (say, a selector that isn't valid) and leaves the stored value alone. Only a selector or regex that matches nothing counts as "no match". An agent taking the profile over mid-read aborts the visit, and nothing is recorded.

  • Changing selector, text_regex, kind or the url starts a new baseline; a new label doesn't. A visit still running for the old configuration records nothing.

• What is stored: only the value (a number, or at most 200 characters), when it was read and when it last changed, shown under watch in keepalive_list, plus baseline / unchanged / changed / lower per visit in last. Nothing else from the page.

The event uses its own MCPL feature set bosun.watch: tags bosun:watch-changed, bosun:source=keepalive, bosun:profile=<name>; origin {profile, label, old, new, url}; text e.g. Bosun watch "willhaben messages" on profile "willhaben" changed: 43 -> 45 (https://…). Delivery and queueing work as for bosun:needs-human.

Host recipe change: a recipe without enabledFeatureSets for Bosun gets both feature sets. One that lists "bosun.handoff" must add "bosun.watch", or use "bosun.*", otherwise watch events are not delivered (they wait in the queue, up to 20 per tenant, until a connection grants the set):

"mcpServers": { "bosun": { "url": "wss://bosun.example/mcp", "transport": "websocket",
                           "enabledFeatureSets": ["bosun.handoff", "bosun.watch"] } }

Limits: a watch sees only what that one page shows (the visible text, or a selector's text, at visit time). Between visits nothing is watched, so with every_minutes: 30 a new message is reported about 15 minutes after it arrived on average and up to about 40 minutes after (longer in quiet hours). Several changes between two visits make one event, carrying the latest value.

Meetings (PoC)

Bosun can attend a Google Meet as a notetaker: it joins as an unauthenticated guest, or with a Bosun profile that is signed in to a Google account, records the meeting's audio and leaves. Nothing is transcribed live: the agent gets the recording when the meeting ends and transcribes it afterwards with whatever it likes. Zoom and Teams are not supported yet; the Meet-specific parts live in src/meetings/platforms/meet.mjs behind a small platform interface, so another platform is one more file.

meeting_join({ "url": "https://meet.google.com/abc-defg-hij" })   // name, announce_text, max_wait_admit_min (10),
                                                                   // profile, engine, leave_on_optout (true)
//  -> { "meeting_id": "m1a2b3c4d5e6", "state": "waiting", ... }   once the join request is out
meeting_status({ "meeting_id": "m1a2b3c4d5e6" })   // joining | waiting | admitted | recording | ended, participants, recording_seconds
meeting_chat_read({ "meeting_id": "…", "since": 0 })      meeting_chat_send({ "meeting_id": "…", "text": "…" })
meeting_stop_recording({ "meeting_id": "…", "delete": true })   meeting_leave({ "meeting_id": "…" })
meeting_recording_delete({ "meeting_id": "…" })

How a meeting runs:

1. Browser: a fresh browser of its own, or profile (Chromium or patchright only; camoufox is refused).

   • Media permission prompts are accepted (--use-fake-ui-for-media-stream), but no fake devices are used: the microphone is a silent audio source and there is no camera.

   • The browser asks for Meet's English UI (hl=en, Accept-Language en-US). Since the account or region can still win, every text and label is also recognized in German, Slovak, Ukrainian and Russian.

2. Lobby:

   • Microphone and camera must be verifiably off before joining. Bosun switches a device off and checks it, unless Meet shows the device doesn't exist ("Camera not found" / "Kamera nicht gefunden") and no control shows it on; then it already counts as off. If neither can be confirmed, Bosun does not join.

   • A guest types its name (default Glassy (notetaker)); a signed-in profile has no name field and joins under the account's name.

   • Then Ask to join / Teilnahme erbitten / Join now, and Bosun waits up to max_wait_admit_min.

3. Not getting in:

   • A denied request or no admission in time ends the meeting with a clear error.

   • Google turning the guest away ("You can't join this video call" / "Sie können nicht an diesem Videoanruf teilnehmen" after asking, with no waiting room seen) ends with reason refused_by_google.

   • That refusal comes with a hint: join with a signed-in profile, or from a residential IP. It is not a captcha, and navigate reports it as a refusal too.

4. Once admitted, Bosun opens the chat and posts the announcement first. Only then does recording start. The default text is in English and German: "Hi, I'm Glassy, an AI notetaker. I'm recording audio for notes. If you'd rather not be recorded, just say so here in chat (e.g. 'please don't record') and I'll stop and delete the recording." With MEETING_OWNER set it says whose notetaker it is.

5. While in the call:

   • Every chat message from someone else is pushed to the agent as bosun:meeting-chat, with meeting_id, sender and text. meeting_chat_read returns them too.

   • Bosun leaves by itself when the meeting ends, when it is removed, or after 2 minutes alone.

   • The agent can leave with meeting_leave.

   • On leaving, bosun:meeting-ended carries meeting_id, reason, duration_s, bytes and the recording: path, a file:// URI and size. The payload also has it as an MCP resource_link.

Signed-in profile. Google turns away anonymous automated guests far more often than signed-in users.

• Setup: session_open({ "profile": "google" }), navigate to https://accounts.google.com, request_human so a person signs in, session_close.

• Joining: meeting_join({ "url": "…", "profile": "google" }).

• Engine: the profile is opened with the engine it was last used with. It can't be in use by an agent session at the same time.

The meeting runs in a session of its own, so it outlives the agent's connection. It counts against MAX_SESSIONS. Events use the MCPL feature set bosun.meeting (tags bosun:meeting-chat, bosun:meeting-optout, bosun:meeting-ended); host recipes that list enabledFeatureSets need to add it (or bosun.*).

Consent. Recording starts only after the announcement. No slash commands are needed: a plain-language request not to be recorded is enough, in English, German, Ukrainian, Russian or Slovak (please don't record, stop recording, bitte nicht aufnehmen, не записуй, не записывай, nenahrávaj, …).

• What happens: Bosun stops the recording and deletes the file at once, posts "Recording stopped and deleted. / Aufnahme gestoppt und gelöscht." in the chat, sends bosun:meeting-optout and leaves. With leave_on_optout: false it stays but never records again in that meeting.

• The matcher is conservative. "don't stop recording", "keep recording" or "record the next steps" do not trigger it.

• Because it can miss phrasings, every chat message also reaches the agent, which can call meeting_stop_recording itself.

Recordings: BROWSER_DATA/recordings/<tenant>/<meeting_id>.ogg, mode 0600, Ogg/Opus 16 kHz mono 24 kbps (about 11 MB per hour). They are deleted after RECORDING_RETENTION_DAYS (default 30), or with meeting_recording_delete. Chat text is kept only in memory, for the last 50 meetings.

Audio setup (Linux). Each meeting gets a private PulseAudio server: its own XDG_RUNTIME_DIR, socket and config, with two null sinks:

• bosun_out: the browser plays the meeting into it, and ffmpeg records its monitor.

• bosun_mic: its silent monitor is the browser's microphone.

Meetings never hear each other or the host's sound card. The image needs:

apt-get install -y pulseaudio pulseaudio-utils ffmpeg

Each meeting starts its own pulseaudio process, so PipeWire alone (pipewire-pulse) isn't enough. Both can be installed side by side: the private servers don't touch the system one.

Meetings need a headful browser (headless Chromium plays no audio), so run the service under Xvfb without HEADLESS=1. On macOS or without these packages, meeting_join refuses with the reason. docker/meeting-test.Dockerfile builds an image that runs the audio test: docker run --rm --init -v "$PWD":/app -w /app bosun-meeting-test xvfb-run -a node --test test/meeting-audio.test.mjs.

Limits — what this can't do (yet):

• Guests need a host. Someone in the meeting has to admit the bot. Meetings that don't allow guests, or that only admit people from the organisation, can't be joined.

• Google reviews bot-like guests. Join requests from automated browsers can land in a review queue or be refused outright ("You can't join this video call", reason refused_by_google). Datacenter IPs make this much more likely. A signed-in profile or a residential connection works best.

• Meet's markup changes without notice. Labels and texts are matched in English, German, Slovak, Ukrainian and Russian (English is requested); the chat sender name is a best-effort guess. A change on Google's side breaks the join or the chat watch until platforms/meet.mjs is updated. This PoC has been tested against a local Meet stand-in, not against live Meet.

• One mixed audio track. There is no speaker separation and no live transcript.

• Consent is the operator's responsibility. The announcement and the opt-out are a courtesy, not a legal basis. Check what applies where the participants are; some places require everyone's explicit consent before recording.

Security notes

• One service is one trust domain per tenant (token): a tenant's connections share its persistent profiles; sessions are private to the connection. For hard isolation run one process (and data dir) per tenant.

• Web content is untrusted (prompt injection). Run the service where it can't reach private networks (block 10/8, 172.16/12, 192.168/16, 100.64/10, 169.254/16 egress) — otherwise a page can steer it at your LAN. The http(s)-only check on navigate does not replace that boundary. Run it as a user that can't read secrets.

• Live-view links are bearer tokens for the whole handoff (up to HANDOFF_TTL_SEC): anyone holding the link can open the live view whenever nobody else is connected, or 5 s after the connected viewer went silent. Serve them on a private network (VPN/mesh) or behind TLS, send them over a private channel, and press Done or Give up when finished — that revokes the link for good (no replay afterwards). A connected, responsive viewer can't be pushed out by a second one.

• No rate limiting yet beyond the connection, in-flight and session caps.

Roadmap

M1 meetings beyond the PoC (Zoom, Teams, speaker separation), tenant quotas/audit log, mobile viewport for handoff, more push events (handoff done/timeout) · M2 vault (fill_credential), downloads as content references, watching pages outside keepalive visits · M3 macros.

License

Apache License 2.0 — see LICENSE and NOTICE.

Status: early preview (0.x). APIs, tool names and config may change without notice.