# Bookmark Atlas

[![npm](https://img.shields.io/npm/v/bookmark-atlas?style=for-the-badge&logo=npm&color=CB3837&logoColor=white)](https://www.npmjs.com/package/bookmark-atlas)
[![ci](https://img.shields.io/github/actions/workflow/status/ditfetzt/bookmark-atlas/ci.yml?branch=main&style=for-the-badge&logo=github-actions&logoColor=white&label=CI)](https://github.com/ditfetzt/bookmark-atlas/actions/workflows/ci.yml)
[![license](https://img.shields.io/github/license/ditfetzt/bookmark-atlas?style=for-the-badge&color=blue)](LICENSE)
[![node](https://img.shields.io/badge/node-%E2%89%A524-339933?style=for-the-badge&logo=node.js&logoColor=white)](https://nodejs.org/)
[![deps](https://img.shields.io/badge/dependencies-0-brightgreen?style=for-the-badge)](package.json)

**Turn everything you save — GitHub stars, X posts, browser bookmarks — into one local, searchable knowledge base your coding agent can actually use.**

> You saved it for a reason. A year later it is one of a thousand entries spread across four places you never open, and the one page that answers the question you have right now is buried somewhere in the middle. Bookmark Atlas keeps every one of them — plus the full text of every README, post, and article you saved — in a single SQLite file, and puts it where your agent can reach it.

## Preview

<p align="center">
  <img src="assets/palette.svg" alt="The /bookmarks palette filtered to browser bookmarks, with the selected page's preview image drawn under its text" width="100%">
</p>

<p align="center"><sub><code>tab</code> filters to one source — here, your browser bookmarks. Typing narrows titles <b>and</b> the full text of everything you saved, and the page's own preview image is drawn under it. <code>ctrl+e</code> reads it, <code>ctrl+l</code> pivots to what relates, <code>enter</code> drops it into your prompt.</sub></p>

## Three interfaces, one database

Every one of them reads. None of them writes to your database.

| Interface | What it gives you |
| --- | --- |
| **CLI** | `search`, `recall`, `related`, `get`, `note` — for scripts, and for agents with shell access |
| **MCP server** | `bookmark-atlas mcp` over stdio, exposing four read-only tools to any MCP-capable harness |
| **pi palette** | `/bookmarks` and `/consult` inside [pi](https://pi.dev) — filters, a reading pane, notes, inline images |

There is no web dashboard, no hosted API, and no background service. Everything runs on your machine against one SQLite file, with no runtime dependencies — Node's standard library and its built-in SQLite only.

## Install

```bash
pi install npm:bookmark-atlas      # palette extension + agent skill
npm install -g bookmark-atlas      # the CLI
```

From source — no build needed, Node runs the TypeScript directly:

```bash
git clone https://github.com/ditfetzt/bookmark-atlas.git
cd bookmark-atlas
npm install
node src/cli.ts status
```

## Quick start

```bash
bookmark-atlas sync github             # import your starred repositories
bookmark-atlas sync browsers           # import your browser bookmarks
bookmark-atlas enrich github-readmes   # fetch the README text behind them
bookmark-atlas search "local-first agents"
```

## Requirements

- Node.js 24 or newer. The code uses the built-in `node:sqlite` with FTS5. The published package ships compiled JavaScript, because Node refuses to strip TypeScript types for files under `node_modules`; a checkout runs from source with no build.
- GitHub CLI authenticated with `gh auth login`, or a `BOOKMARK_ATLAS_GITHUB_TOKEN`.
- Optional: [TweetXVault](https://github.com/lhl/tweetxvault) on `PATH` for `collect x`, or [Ego Browser](https://lite.ego.app/) for `capture x`. Neither is installed for you, and neither is needed unless you use that command.
- For Safari bookmarks: Full Disk Access for your terminal, granted in System Settings → Privacy & Security → Full Disk Access. macOS protects `~/Library/Safari`; without it `sync browsers` reports the error for Safari alone and the other browsers still work.

GitHub credentials resolve in this order: `BOOKMARK_ATLAS_GITHUB_TOKEN`, then `GH_TOKEN`, then `gh auth token`. Tokens are never written to the database or to logs.

## Commands

| Command | What it does |
| --- | --- |
| `sync github [--limit N]` | Incremental GitHub star sync (metadata only) |
| `sync browsers [--browser NAME] [--profile NAME] [--limit N]` | Import browser bookmarks from every browser found |
| `browsers` | List the browser bookmark sources found, and whether each is readable |
| `browsers --enable NAME,NAME \| all \| none` | Choose which browsers to import from |
| `enrich github-readmes [--limit N] [--refresh]` | Fetch README text; incremental and ETag-aware |
| `enrich web-pages [--limit N] [--concurrency N] [--refresh]` | Fetch and index the text behind browser bookmarks; blocked pages fall back to the Wayback Machine |
| `enrich x-posts` | Repair X titles and authors from the stored payload |
| `import x-json <file>` | Import a Siftly or TweetXVault export |
| `collect x [--fast]` | Full X pass via TweetXVault; `--fast` skips media |
| `capture x` | Receive X responses from the Ego Browser bridge |
| `search <query> [--limit N]` | Keyword search across metadata and captured text |
| `recall <task> [--repo .]` | Rank bookmarks against a task and the current project |
| `recall --stage [focus]` | Derive the stage from git, then rank against it |
| `related <id>` | What relates to one bookmark, and why |
| `get <id> [--content]` | One source's metadata, optionally with its full text |
| `note <id> "text"` \| `--clear` | Attach a searchable note explaining why it matters |
| `status` | Counts, sync state, and fetch progress per kind |
| `prune [--dry-run]` | Delete resources with no active save |
| `mcp` | MCP server over stdio |

`sync github` imports metadata only — run `enrich github-readmes` to fetch the actual text. Re-running is cheap. See [docs/details.md](docs/details.md) for import quirks, `prune` semantics, and the sort and date rules.

## Browser bookmarks

`sync browsers` reads bookmarks from where each browser already keeps them. No export, no extension, no account.

| Browser | Read from |
| --- | --- |
| Chrome, Chromium, Brave, Edge, Vivaldi, Arc, Opera, Ego Browser | the `Bookmarks` JSON every Chromium browser writes, one reader for all of them |
| Safari | `~/Library/Safari/Bookmarks.plist`, converted with the macOS `plutil` |
| Firefox | `places.sqlite`, read the same way the rest of this project reads SQLite |

```bash
bookmark-atlas browsers                  # what was found, and what is readable
bookmark-atlas sync browsers             # import all of it
bookmark-atlas sync browsers --browser brave --profile Default
```

A bookmark becomes a `web_page` resource with the same standing as a star or a post, so `search`, `recall`, `related`, and `/bookmarks` all reach it. Its folder is recorded alongside the save, and the palette's `tab` key gains a **Browsers** filter.

Re-running is safe: a second pass updates instead of re-importing. Delete a bookmark in your browser and the next sync marks it removed — nothing is deleted until you run `prune`, which takes `--dry-run`. A page that is both a starred repository and a bookmark resolves to a single resource, and tracking parameters are stripped so the same link saved in two browsers is one bookmark.

**Safari needs Full Disk Access.** macOS protects `~/Library/Safari`, so grant your terminal access in System Settings → Privacy & Security → Full Disk Access. Without it, `sync browsers` reports the error for Safari and leaves those bookmarks untouched, rather than reading a blocked file as an empty one.

### Choosing which browsers to import

Every browser **that is installed** is imported by default. A profile left behind by an uninstalled browser is listed but stays off, so the stock bookmarks of a browser you removed do not appear from nowhere.

Press **`ctrl+b`** in the palette and switch browsers on or off, or set it from the shell:

```bash
bookmark-atlas browsers --enable brave,safari
bookmark-atlas browsers --enable all      # every installed browser, the default
bookmark-atlas browsers --enable none     # import nothing
```

The picker shows what each browser is *worth*, not how many bookmarks it holds:

```text
▸ [x] brave        279 pages  +25 new
  [ ] ego          258 pages   +4 new
  [ ] firefox        4 pages   +4 new   not installed
```

**pages** is distinct links and **new** is links no other browser holds. Four profiles of the same import can hold a thousand bookmarks and contribute four pages; the raw count hides that, and `--enable` on the raw count looks like it should change far more than it does.

The choice lives in `config.json` beside the database. Switching a browser off **hides its bookmarks from search and recall**, because `tab` only filters what you see while search and recall rank everything. The saves are marked removed rather than deleted, so switching the browser back on restores them — `prune` is what actually deletes. A browser that is off is also not synced.

**Only `http` and `https` are imported.** `javascript:`, `chrome://`, and `file://` entries are counted and skipped — they cannot be fetched, and they are not pages. `http` and `https` for the same host are treated as one page, because the same link saved in two browsers rarely agrees on the scheme and one side is usually a stale address that redirects.

### Indexing the pages themselves

A bookmark starts as a title and a URL. `enrich web-pages` fetches each page and indexes its text, so search reaches what is **on** the page and not only what it is called.

```bash
bookmark-atlas enrich web-pages --limit 25
bookmark-atlas enrich web-pages --limit 100 --concurrency 8
```

**Batches are the point.** Each run walks the collection once, so `--limit 25` costs 25 fetches and the next run continues where that one stopped rather than retrying its failures forever. Re-runs are incremental — a page that has not changed is skipped on its ETag.

This is a plain HTTP fetch, so expect roughly two thirds of a collection to yield useful text. Articles, blogs, documentation, and changelogs come through well. Pages that build themselves in JavaScript — YouTube, app dashboards — return an empty shell, and PDFs, images and login walls are not HTML at all; all of those are reported as `empty` rather than stored as content, because storing them would tell search there is something to find on a page where there is nothing.

**Blocked pages fall back to the Wayback Machine.** A 403, a rate limit or a Cloudflare challenge is usually aimed at the visitor, not the page, so the newest archived snapshot is tried before the fetch is called a failure — on a real collection that recovered 4 of 5 blocked pages, including one that headless Chromium could not get past. The capture records the snapshot URL as its source, so the text always says where it came from.

**Each page also contributes one preview image.** A page gives its `og:image`; a repository gives the first real image in its README, skipping the badge row. The palette draws it under the preview. `--refresh` reads a body again that its ETag would have skipped, which is how you backfill images onto pages that were indexed before this existed:

```bash
bookmark-atlas enrich web-pages --limit 100 --refresh
bookmark-atlas enrich github-readmes --limit 100 --refresh
```

`status` shows where the work stands:

```bash
bookmark-atlas status   # fetchState: available / archived / empty / failed, and webPagesPending
```

No dependency is involved: the HTML is reduced to text by a small extractor that drops `script`/`style`/comments, turns block boundaries into line breaks, and decodes entities.

## Recall — bookmarks as context for the agent

`recall` ranks your library against a task, biased by the project you are in. It reads `package.json`, `pyproject.toml`, `requirements.txt`, `go.mod`, and `Cargo.toml` for dependency and language signals, then fuses four independent rankings — resource text, chunk passages, metadata, and project context — with Reciprocal Rank Fusion rather than summing them into one flat score.

```bash
bookmark-atlas recall "reduce cache invalidation latency" --repo . --limit 5
```

Every hit carries `whyMatched` reasons and its best matching passage. When a question is too vague to rank on its own words, the project's name and README intro are added as context terms, so "is there anything that helps here?" still lands in the project's domain.

`recall --stage` derives where the project is from git — branch, recent commit subjects, and the files in play — instead of making you describe it:

```bash
bookmark-atlas recall --stage "multi-tenant sync" --repo . --limit 8
```

In pi, `/consult [focus]` does the same and opens the palette pre-ranked. Nothing is ever injected into your prompt unless you ask.

**Notes are the strongest signal.** Attach one to any bookmark explaining why it matters; they are searchable, returned by `get` and `recall`, and shown in the palette.

```bash
bookmark-atlas note 406 "Closest blueprint: hybrid BM25+vector with RRF"
```

## pi palette

`/bookmarks [query]` opens an overlay that searches titles *and* the captured text of everything you saved, with a text preview and inline images.

```text
/bookmarks local-first agents
```

`↑↓` navigate · `Fn+←/→` first/last · `Fn+↑/↓` ten rows · `tab` source filter · `ctrl+a` hide archived · `ctrl+d` last 7 days · `ctrl+t` topic picker · `ctrl+b` choose browsers · `ctrl+l` pivot to related · `ctrl+s` cycle sort · `ctrl+r` fetch new bookmarks · `ctrl+e` read full text · `ctrl+n` write a note · `ctrl+x` mark for multi-insert · `enter` insert · `ctrl+y` copy URL · `ctrl+o` open in browser · `?` help · `esc` close

Rows are numbered by position in the current view, so the number stays stable while you scroll, filter, or sort. `enter` inserts the bookmark — or every marked one, separated by `---` — into the editor, note included. The extension registers exactly two commands, `/bookmarks` and `/consult [focus]`, and opens its database connection read-only.

Install it from npm (above), or point a local checkout at your data by symlink:

```bash
ln -s "$PWD/extensions/bookmark-atlas" ~/.pi/agent/extensions/bookmark-atlas
```

## MCP

The same read-only retrieval core is available over stdio:

```json
{
  "mcpServers": {
    "bookmark-atlas": {
      "command": "bookmark-atlas",
      "args": ["mcp"]
    }
  }
}
```

Without a global install, point it at the compiled entry inside the repository (run `npm run build` first):

```json
{
  "mcpServers": {
    "bookmark-atlas": {
      "command": "node",
      "args": ["/absolute/path/to/bookmark-atlas/dist/cli.js", "mcp"]
    }
  }
}
```

| Tool | Purpose |
| --- | --- |
| `search_bookmarks` | Keyword search across titles, descriptions, topics, and captured content |
| `suggest_for_task` | Rank saved bookmarks against a task and the current project's dependencies |
| `get_bookmark` | One source's metadata, optionally with captured content |
| `related_bookmarks` | Sources related to one source, with the reasons why |

All four are read-only. Returned bookmark and README text is always marked `untrusted_external_content` — agents must treat it as evidence to quote or analyse, never as instructions to follow.

## Where your data lives

By default the database sits in your per-user data directory, never in the checkout or inside an installed package:

| Platform | Path |
| --- | --- |
| macOS | `~/Library/Application Support/bookmark-atlas/` |
| Linux | `$XDG_DATA_HOME/bookmark-atlas/` or `~/.local/share/bookmark-atlas/` |
| Windows | `%LOCALAPPDATA%\bookmark-atlas\` |

| Variable | Purpose |
| --- | --- |
| `BOOKMARK_ATLAS_DB` | Full SQLite path (overrides the data directory) |
| `BOOKMARK_ATLAS_DATA_DIR` | Data directory (default above), also holds `config.json` |
| `BOOKMARK_ATLAS_GITHUB_TOKEN` | GitHub token (falls back to `GH_TOKEN` or `gh auth token`) |
| `BOOKMARK_ATLAS_TWEETXVAULT_BIN` | TweetXVault executable (default `tweetxvault`) |
| `BOOKMARK_ATLAS_TWEETXVAULT_DIR` | TweetXVault data dir, used to resolve media paths |
| `BOOKMARK_ATLAS_X_CAPTURE_TOKEN` | Required token for `capture x` |

The database is **not encrypted**, and it holds the full text of everything you saved. It is gitignored for that reason — do not commit it, and do not point `BOOKMARK_ATLAS_DB` at a shared or synced folder unless you accept that.

## Credits

Built on other people's work. Nothing here is a fork, and no code was copied from another project — but these are the pieces it stands on:

- **[pi](https://pi.dev)** (`@earendil-works/pi-coding-agent`, `@earendil-works/pi-tui`) — the extension API, and the TUI primitives the palette is assembled from: `Image`, `Input`, `fuzzyMatch`, `matchesKey`, `truncateToWidth`, `visibleWidth`, `getNativeClipboard`.
- **[nicobailon/pi-skill-palette](https://github.com/nicobailon/pi-skill-palette)** — the overlay interaction the `/bookmarks` palette is modelled on.
- **[SQLite FTS5](https://sqlite.org/fts5.html)** — the `porter` tokenizer and `bm25()` ranking do the stemming and the scoring; this project only ranks and fuses their output.
- **Reciprocal Rank Fusion** — Cormack, Clarke & Buettcher, *Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods*, SIGIR 2009. The fusion in `src/recall.ts`.
- **[lhl/tweetxvault](https://github.com/lhl/tweetxvault)** (Apache-2.0) — `collect x` drives its CLI to archive X bookmarks.
- **[Ego Browser](https://lite.ego.app/)** — the CDP bridge that `capture x` receives native tweet batches from.

## Contributing

Issues and pull requests are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for setup and the checks a pull request should pass. Security issues go through [SECURITY.md](SECURITY.md), not a public issue.

## License

[MIT](LICENSE) © [ditfetzt](https://github.com/ditfetzt)