<div align="center">

# Subio Scrape

**Read what people are actually posting — in a browser that is never signed in.**

An MCP server that opens a browser on your own machine, reads the public posts a
signed-out visitor is shown on X, LinkedIn and Hacker News, and hands them back
as structured data your agent can reason about.

[Install](INSTALL.md) · [Tools](#tools) · [What signed-out access buys you](#what-signed-out-access-actually-buys-you) · [Blocks](#blocks-and-what-to-do-about-them) · [Security](SECURITY.md)

</div>

---

## What this is

Ask your agent *"what have @naval and @paulg been posting about onboarding this
week?"* and it opens a real browser window, reads those two timelines, and
answers with the posts, their engagement counts and their links.

Everything happens locally. There is no API key, no account, no cloud service and
no data leaving your machine except the page requests themselves.

**It never signs in.** Session cookies are deleted from its browser profile
before every run. It reads exactly what a logged-out visitor is shown — nothing
behind a login, nothing behind a challenge. It does not solve CAPTCHAs, does not
disguise that it is automated, and does not retry a refusal.

That constraint is the point, and it shapes what the tool can and cannot do. The
sections below are honest about both.

## Quick start

```bash
claude mcp add subio-scrape -- npx -y github:alijancb/subio-mcp
```

Then ask, in plain language:

> собери, что @naval и @paulg писали за неделю про онбординг

Using a different agent — Codex, Cursor, Windsurf, Cline, Zed, VS Code? See
**[INSTALL.md](INSTALL.md)**, which has a one-page guide per client that your
agent can follow for you.

For Claude Code there is also a plugin, which adds a `/scrape` command and a
workflow skill on top of the tools:

```bash
git clone https://github.com/alijancb/subio-mcp.git ~/subio-mcp && cd ~/subio-mcp && npm install
```

```bash
claude plugin marketplace add ~/subio-mcp && claude plugin install subio-scrape@subio
```

### Requirements

- **Node.js 20 or newer.**
- **A browser.** It uses Chrome or Edge if you have one. If not, run
  `npx playwright install chromium` once.
- **A desktop session.** The window has to be visible — see
  [below](#what-signed-out-access-actually-buys-you) for why. This does not run
  on a headless server.

## Tools

| Tool | What it reads |
| --- | --- |
| `x_account_posts` | Posts from named X accounts, with exact view / like / repost / reply counts. `match` filters them by keyword or hashtag |
| `x_account_profile` | Bio, followers, post count for one account |
| `linkedin_page` | A public company or person page, from its JSON-LD |
| `linkedin_company_posts` | A company's posts, when LinkedIn serves them signed out |
| `hackernews_search` | Hacker News stories and comments on a topic |
| `browser_status` | Which browser will be driven, from where, how fast, and what happens on a block |
| `browser_close` | Close the window early |

Every tool returns the same item shape, so results from four sources merge into
one report:

```json
{
  "source": "x",
  "externalId": "1960...",
  "url": "https://x.com/naval/status/1960...",
  "author": "naval",
  "text": "…",
  "publishedAt": "2026-08-27T02:22:11.000Z",
  "engagement": { "views": 8499257, "likes": 14021, "replies": 689, "reposts": 1367 },
  "via": "@naval"
}
```

### Results tell you what actually happened

Collector tools answer with `{ items, notes, partial }`, and **`notes` is not
decoration**. A source that was blocked, walled or misspelled returns zero items
— and so does a genuinely quiet week. Without the notes, an agent reports the
second for both:

```
notes: [
  "@naval: 2 items",
  "@paulg: 2 items",
  "@definitely-not-real: FAILED — not-found: X sent @definitely-not-real to its
   signup page instead of a profile. Either the handle does not exist, or X
   declined to serve it signed out."
]
partial: true
```

## What signed-out access actually buys you

Worth reading before you plan anything around this.

- **X serves account timelines to a signed-out browser, and serves search to
  nobody.** `x.com/search`, hashtags and explore all redirect to a login wall.
  There is deliberately no search tool here, because there is no signed-out
  search. A topic on X is answered by reading the accounts that discuss it and
  filtering with `match`.
- **X answers a headless browser with a bare 403 and an empty document.** The
  window is visible by default for that reason, and not as a preference.
  `SUBIO_MCP_HEADLESS=1` exists and mostly does not work.
- **A missing X handle is not a 404.** X answers 200 and redirects into its
  signup flow, which reads exactly like an empty timeline. The final URL is what
  gives it away, and is checked.
- **LinkedIn gives the least and takes it away fastest.** A public company or
  person page renders, usually with JSON-LD on it. After a handful of pages from
  one client it becomes a sign-in wall — served with a 200 status and carrying an
  `og:title` of its own, so a wall parses as a page named "Sign Up | LinkedIn"
  unless you check for the JSON-LD. Expect to be walled within a session.
- **Hacker News has a real public API** and is the one source with no ceiling.
- **Reddit was removed rather than worked around.** It closed `/*.json` to
  clients without a session, and its public RSS feed carries neither score nor
  comment count, so what remained could not be ranked against anything else.

## Blocks, and what to do about them

**Nothing here retries a refusal.** A 403, a 429, a challenge or a sign-in wall
is the site answering, and asking again is how being throttled becomes being
banned. The first block also ends the whole run: the accounts that were not read
are listed in the notes, with an instruction not to retry them now.

If you are getting blocked, what helps, in order:

1. **Read fewer accounts per run.** This is the whole answer most of the time.
2. **Widen the gap** — `SUBIO_MCP_MIN_INTERVAL_MS=10000`. Reads are already
   jittered by ±25% so they do not arrive on a metronome.
3. **Wait.** A signed-out quota refills on its own, and nothing you change
   locally makes it refill faster.

What does **not** help is a fresh browser per account. Blocks are counted against
your IP address, and a new browser profile does not change your address — a
hundred brand-new fingerprints from one address is a stronger bot signal than one
returning visitor. `SUBIO_MCP_ISOLATE=1` exists if you want to try anyway: it
opens a throwaway browser on a throwaway profile per account and deletes it
afterwards. It costs a browser launch per account, and it resets the
cookie-scoped guest quota — which is working around a limit rather than
respecting one, and is the opposite of every other decision here. Off by default.

## One window, one tab

A scan does not open a browser per account. The server keeps a single browser
with a single tab and drives every account, company and page through it, one load
at a time — lighter on your machine than a tab per source, and less conspicuous
to the sites being read.

The window closes itself five minutes after the last read, and immediately if the
client goes away, so nothing is ever left running with nobody driving it.

## Configuration

Everything has a working default. Set these only if you need to.

| Variable | Default | |
| --- | --- | --- |
| `SUBIO_MCP_PROFILE` | `~/.subio-mcp/browser-profile` | Browser profile, kept apart from your own Chrome |
| `SUBIO_MCP_CACHE` | `~/.subio-mcp/cache` | Read cache |
| `SUBIO_MCP_CACHE_TTL_MS` | `1800000` | How long a read counts as fresh |
| `SUBIO_MCP_MIN_INTERVAL_MS` | `4000` | Minimum gap between two loads of one host |
| `SUBIO_MCP_IDLE_MS` | `300000` | Close the window after this long idle. `0` never |
| `SUBIO_MCP_HEADLESS` | unset | `1` runs without a window. X will refuse |
| `SUBIO_MCP_BROWSER` | unset | Force `chrome` or `msedge`, or leave it to pick |
| `SUBIO_MCP_ISOLATE` | unset | `1` opens a throwaway browser per account. Read [above](#blocks-and-what-to-do-about-them) first |

Session cookies (`auth_token`, `ct0`, `li_at`, …) are deleted from the profile
before every run, so the promise on the tin holds even if you once opened that
profile by hand and signed in.

## Development

```bash
npm install
```

```bash
npm run check && npm run build && node scripts/probe.mjs
```

`scripts/probe.mjs` starts the built server over stdio the way a client does and
exercises the tools that need no browser. It catches a broken build, a bad tool
schema, or anything accidentally written to stdout — all of which look like a
silent hang from inside an agent. `scripts/live.mjs` is the manual counterpart
that reads the real sites; it is not in CI, because CI should not scrape.

Where things live:

| Path | |
| --- | --- |
| `src/server.ts` | The tool surface |
| `src/browser/session.ts` | One window, pacing, signing out, idle close |
| `src/collect/` | One module per source |
| `src/parse/x.ts` | Everything known about X's markup — the only file to edit when X changes |
| `src/parse/linkedin.ts` | The same for LinkedIn, parsed as HTML in node |
| `skills/`, `commands/` | The Claude Code plugin's workflow |

The X parser runs inside the page. The LinkedIn parser deliberately does not:
LinkedIn redirects on entry, and anything evaluated in the document loses its
execution context when it does. It reads the HTML as a string instead, which
makes it immune to that rather than lucky about it — and testable from a fixture.

Contributions welcome: see [CONTRIBUTING.md](CONTRIBUTING.md).

## History

This repository previously held a different Subio MCP server — one that brought a
live Subio workspace into MCP clients over the Subio API. It was replaced rather
than extended, because a local signed-out scraper and an API client share a name
and nothing else. That work is preserved on the
[`legacy/subio-api-server`](https://github.com/alijancb/subio-mcp/tree/legacy/subio-api-server)
branch and under the `v0.1.0` tag.

## License

[MIT](LICENSE).