# AI Papers Helper

> Search, index, and read academic AI/ML papers — from the terminal or directly inside your AI coding assistant.

AI Papers Helper crawls top-tier AI/ML conference proceedings, indexes them in a local SQLite database with FTS5 search, queries arxiv, and converts paper PDFs into clean markdown for in-depth reading. It ships as both a **CLI** (`papers`) and an **MCP server** (`ai-papers-mcp`), so you can use it standalone or wire it into Claude Code / any MCP-compatible client.

---

## ✨ Features

- **🌐 Conference crawling** — Fetch papers from NeurIPS, ICML, ICLR, CVPR, ICCV, WACV, AAAI, IJCAI, MLSys, ACL, EMNLP with parallel workers and incremental crawl state.
- **🔎 Local search** — SQLite + FTS5 with BM25 ranking (title-weighted) and relevance/date ordering. Missing-abstract penalty keeps results meaningful.
- **📡 arxiv search** — Live arxiv search via web scraping (default) or the Atom API backend, with date filters (past 12 months, specific year, date range).
- **📄 PDF → Markdown** — Parse papers through the [MinerU](https://mineru.net) API with content-addressable caching, so each PDF is only parsed once.
- **📑 TOC & grep** — Extract a paper's table of contents and `grep` specific sections/equations from its full markdown.
- **🤝 MCP server** — Expose everything as 4 tools to Claude Code or any MCP client: search the local library, search arxiv, get a paper's TOC, and grep paper content.
- **🧠 Smart resolution** — Paper lookup falls back progressively: exact DB match → fuzzy match (`SequenceMatcher`, 0.8 threshold) → arxiv search.

---

## 📦 Installation

Requires **Python 3.12+**.

```bash
# Clone
git clone https://github.com/<your-org>/ai_papers_helper.git
cd ai_papers_helper

# Install as a global tool (exposes `papers` and `ai-papers-mcp` on PATH)
uv tool install --force .
```

This exposes two console scripts:

| Command         | Description                          |
| --------------- | ------------------------------------ |
| `papers`        | The CLI app                          |
| `ai-papers-mcp` | The MCP server                       |

### Environment variables

| Variable              | Default | Description                                                              |
| --------------------- | ------- | ------------------------------------------------------------------------ |
| `MINERU_API_KEY`      | —       | **Required** for PDF→markdown parsing. Get one at mineru.net.            |
| `ARXIV_BACKEND`       | `web`   | arxiv backend: `web` (scraping) or `api` (Atom API).                     |
| `ARXIV_MIN_INTERVAL`  | `10.0`  | Minimum seconds between arxiv API calls (rate limiting).                 |

All data lives under `~/.ai_papers_helper/`:

```
~/.ai_papers_helper/
├── papers.db            # SQLite database (FTS5 index)
├── crawl_state.json     # Incremental crawl state
├── cache/               # HTTP response cache
├── arxiv_cache/         # arxiv result cache (24h TTL)
└── papers/              # Parsed markdown (content-addressed by URL hash)
```

---

## 🚀 Quick start

### 1. Initialize the database

```bash
papers init
```

### 2. Crawl conference papers

```bash
# Crawl everything new (incremental — skips already-crawled years)
papers update

# Crawl a specific conference and year
papers update --conference acl,emnlp --year 2024

# Force re-crawl a specific source/year
papers update --force --conference cvpr --year 2023

# Tune parallelism
papers update --workers 16
```

### 3. Search

```bash
# Search the local library (default: titles only)
papers search-library "diffusion model"
papers sl "graph neural network" --order-by date

# Titles + full abstracts
papers sl "transformer attention" --full-abs

# Paginate
papers sl "reinforcement learning" --page 2
papers sl "reinforcement learning" --from 31

# Search arxiv
papers search-arxiv "mixture of experts"
papers sa "vision transformer" --sort-by date --date-filter-by past_12
papers sa "llm" --date-filter-by specific_year --date-year 2024
papers sa "diffusion" --date-filter-by date_range --date-from 2024-01 --date-to 2024-06
```

### 4. Read a paper

```bash
# Get the full markdown (creates a /tmp/<title>.md symlink to the cached file)
papers content "Attention Is All You Need"

# Show the table of contents
papers content "Attention Is All You Need" --toc
```

---

## 🧩 Skill (for AI coding agents)

The repo ships a ready-made [Agent Skills](https://agentskills.io) skill at [`papers-skill/SKILL.md`](./papers-skill/SKILL.md). It teaches AI coding agents ([pi](https://pi.dev/), Claude Code, etc.) how to drive the `papers` CLI: when to prefer the local library vs arxiv, the `search -> content -> grep` workflow, and the full option reference.

The agent loads the skill on-demand when a task matches, then runs `papers` itself via the shell - no server process required. This is the lightest-weight way to let an agent search and read papers.

### Install the skill

Point your agent at the `papers-skill` directory. For pi:

```bash
# Global (available in every project)
ln -s "$(pwd)/papers-skill" ~/.pi/agent/skills/papers

# Or project-level
mkdir -p .pi/skills && ln -s "$(pwd)/papers-skill" .pi/skills/papers
```

> **Skill vs MCP:** The skill is just instructions (the agent runs the CLI via shell); the MCP server below exposes typed tools. The skill needs nothing running, the MCP server gives more structured tool calls - pick whichever fits your agent.

---

## 🤖 Using the MCP server

The same functionality is exposed as an MCP server for use inside Claude Code or any MCP-compatible client.

### 4 tools

| Tool                    | Description                                                                 |
| ----------------------- | --------------------------------------------------------------------------- |
| `search_library_papers` | Search the local indexed database by keywords.                   |
| `search_arxiv_papers`   | Search arxiv for the latest papers, with date filters.                      |
| `get_paper_toc`         | Get a paper's table of contents. **Call this first** before reading.        |
| `grep_paper_content`    | `grep` patterns in a paper's full markdown (e.g. read a whole section).     |

---

## 🏗 Architecture

```
src/ai_papers_helper/
├── cli.py                 # Typer CLI: init, search-library, search-arxiv, update, content
├── mcp_server.py          # FastMCP server exposing 4 tools
├── config.py              # Paths, env vars, page-size constants
├── core/
│   ├── models.py          # Pydantic v2: Author, Paper
│   └── database.py        # SQLite + FTS5 singleton, BM25 ranking, auto-sync triggers
├── crawler/
│   ├── base.py            # BaseCrawler ABC + parallel detail-page fetching
│   ├── http.py            # Shared requests.Session w/ retry + file cache
│   ├── state.py           # CrawlState (per-source crawled years, JSON)
│   ├── cvf.py             # CVPR / ICCV / WACV
│   ├── aaai.py            # AAAI
│   ├── ijcai.py           # IJCAI
│   ├── icml.py            # ICML URL helpers
│   ├── acl_anthology.py   # ACL / EMNLP (ACL Anthology)
│   └── json_api.py        # Generic JSON API crawler (NeurIPS, ICML, ICLR, MLSys)
├── search/
│   ├── library_search.py  # Local FTS5 search, relevance/date ordering
│   └── arxiv_search.py    # arxiv dispatcher (web vs api backend)
├── retrieval/
│   ├── resolver.py        # Progressive lookup: exact → fuzzy → arxiv
│   ├── parser.py          # MinerU API client (async polling, content-addressed cache)
│   ├── content.py         # Markdown TOC extraction + section slicing
│   └── lookup.py          # End-to-end: title → paper → markdown
└── helper/                # arxiv web/api internals, pagination, rate limiting
```

### How search works

- **Local library**: FTS5 with `porter unicode61` tokenizer. BM25 with title weight `10.0`, abstract weight `1.0`. Results with missing abstracts are penalized (`* 0.9`) so well-documented papers surface first.
- **Date ordering**: BM25 rank is bucketed into relevance tiers; within a tier, newer papers come first — so you don't lose relevance entirely.
- **arxiv**: Web scraping by default (no API key, gentler). Switch to the Atom API with `ARXIV_BACKEND=api` for query-syntax power (field prefixes, boolean operators).

### How PDF reading works

1. Resolve the paper by title (DB exact → fuzzy → arxiv).
2. Resolve a PDF URL (`paper.pdf_url`, else arxiv lookup, backfilling the DB).
3. Send to MinerU; poll until `done`; download & unzip the result.
4. Cache under `~/.ai_papers_helper/papers/{sha256(url)[:16]}/full.md` — content-addressed, so re-reads are instant.

---

## 🧪 Development

```bash
uv sync                 # install deps
pytest                  # run the full suite
pytest tests/test_database.py          # single file
pytest -k "fuzzy"                      # by name pattern
```

Tests use temp databases, mock network calls (`patch.object(crawler, "_fetch_url", ...)`), and HTML/JSON fixtures in `tests/fixtures/`. See [`CLAUDE.md`](./CLAUDE.md) for the full contributor guide.

### Conventions

- Python 3.12+, `from __future__ import annotations` in every file
- No async; parallelism via `ThreadPoolExecutor`
- Standard-library `sqlite3` (no ORM), `requests` for HTTP
- Logging via `logging.getLogger(__name__)`

---

## 📄 License

MIT