# Fetchive

<!-- mcp-name: io.github.lsmman/fetchive -->

**Throw links at it. Get workspace-ready pages back.**

An MCP server for people who collect more links than they can process. Paste a
KakaoTalk export, a Slack dump, or a bare list of URLs, and Fetchive pulls out
every link, skips what you already saved, reads what is behind each one —
YouTube transcripts, article text — and hands your assistant a ready-to-save
payload for each.

Everything it reads stays searchable, so last month's links can answer this
month's question.

It does not write to your workspace. It prepares the page and lets whatever you
already have connected — Notion, Obsidian, anything — do the saving. No second
login, no second place for your data to sit.

---

## Install

```bash
uvx fetchive
```

Or add it to your MCP client:

```json
{
  "mcpServers": {
    "fetchive": {
      "command": "uvx",
      "args": ["fetchive"]
    }
  }
}
```

For local speech-to-text on videos with no captions:

```bash
uvx --with 'fetchive[stt]' fetchive
```

That extra needs **ffmpeg** on your PATH (`brew install ffmpeg`) and downloads a
Whisper model on first use.

---

## Using it

Say this to your assistant:

> Here's my link dump from this week, archive it → *(paste)*

Behind that, one tool call does the work:

1. `archive_batch` parses the links, skips what is already saved, extracts
   transcripts, and starts working in the background
2. `get_batch_status` reports progress and returns the finished payloads
3. Your assistant saves each one with your workspace tool
4. `record_archived` tells Fetchive it landed, so next week's dump skips it

Later, ask it what you saved:

> 지난달에 RAG 청킹 얘기 어디서 봤더라?

`search_archive` runs full-text search over every transcript and article body it
has read — not just titles — and Korean works, which the usual word tokenizer
does not manage.

The other tools — `ingest_links`, `extract_youtube_script`, `check_duplicate`,
`format_for_notion`, `forget_archived` — are the individual steps, there for when
you want to inspect or repair something rather than run the whole flow.

**A link it cannot read fails rather than becoming an empty page.** A row that
says only "here is a URL I could not open" looks like the work was done.

---

## How transcripts actually get extracted

This is the hard part of the problem, and no single method survives YouTube for
long. Fetchive tries four, in order, and stops at the first that returns words:

| Order | Method | Cost | Why it is in the chain |
|---|---|---|---|
| 1 | `youtube-transcript-api`, direct | free | Fastest when it works |
| 2 | `yt-dlp` captions | free | Different endpoints, so it survives blocks that stop #1 |
| 3 | `youtube-transcript-api` via proxy | cheap | For when your IP itself is the problem |
| 4 | `yt-dlp` audio → local Whisper | slow | Works on videos with no captions at all |

Every attempt is recorded with a reason it failed — `blocked`, `no_captions`,
`unavailable`, `too_long`, and so on — so a run tells you not only what came out
but which parts of the pipeline still work. Successful transcripts are cached, so
asking twice is free.

Captions are chosen by what exists, not by what you hoped for: a Korean video
with only Korean captions gets its Korean transcript rather than being reported
as having none.

---

## Configuration

Everything is optional.

| Variable | Default | What it does |
|---|---|---|
| `FETCHIVE_DB_PATH` | `~/.fetchive/fetchive.db` | Where your archive index and transcript cache live |
| `FETCHIVE_LANGUAGES` | `ko,en` | Preferred caption languages, best first |
| `WEBSHARE_PROXY_USERNAME` / `_PASSWORD` | — | Enables the proxy step. Without these it is skipped |
| `FETCHIVE_ENABLE_STT` | `true` | Set to `false` to never fall back to speech-to-text |
| `FETCHIVE_WHISPER_MODEL` | `base` | Larger is slower and more accurate |
| `FETCHIVE_MAX_STT_DURATION` | `3600` | Refuse to transcribe anything longer, in seconds |
| `FETCHIVE_BATCH_WORKERS` | `3` | Concurrent extractions. YouTube dislikes more |
| `FETCHIVE_QUOTE_WORD_LIMIT` | `15` | Longest verbatim quote allowed out of the server |
| `FETCHIVE_TRANSPORT` | `stdio` | `http` for a hosted deployment |

---

## About copyright

Transcripts are extracted for your own reading and cached locally. Putting that
text into a stored page is a different act, so what Fetchive prepares for your
workspace is a summary, a word count, and at most one quote of fifteen words per
source — enforced in the payload builder rather than left to whoever is calling.
The link to the original always travels with it.

Fetchive is a tool for organising material you have chosen to keep. It is not for
republishing anyone's work, and you remain responsible for what you do with what
it extracts. If a source's terms do not permit you to store its content, do not
archive it.

---

## Your data

Everything stays on your machine, in one SQLite file. Fetchive has no server, no
account, and no telemetry. It talks to YouTube to fetch transcripts, and to a
proxy only if you configure one. Nothing is ever sent to Fetchive's authors.

To delete everything: `rm -rf ~/.fetchive`. To remove one item, call
`forget_archived` with its URL.

Full policy:
[https://github.com/lsmman/fetchive/blob/main/PRIVACY.md](https://github.com/lsmman/fetchive/blob/main/PRIVACY.md)

---

## Development

```bash
uv sync
uv run pytest
uv run ruff check src tests
```

Network tests are deselected by default. To run them against real YouTube:

```bash
uv run pytest -m network
```

---

MIT licensed.