# Paper Extraction MCP

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io/)

An MCP (Model Context Protocol) server for **structured information extraction from academic PDF papers** using LLM. It integrates seamlessly with Claude Desktop, allowing you to extract metadata and domain-specific content categories from research papers through natural language conversation.

> **Typhoon disaster governance** is provided as a built-in example. The system is fully customizable for **any research domain** — see [Adapting to Other Domains](#-adapting-to-other-domains).

## Features

- **LLM-Powered Extraction** — Uses OpenAI-compatible LLMs to extract structured data from full-text PDFs
- **Customizable Schema** — Define your own metadata fields and content categories via `config.json`
- **Smart Chunking** — Automatically splits long papers and merges results with deduplication
- **Dual Output** — JSON and CSV formats for downstream analysis
- **MCP Protocol** — Works directly inside Claude Desktop as a tool server
- **Batch Processing** — Extract from a single paper or all papers at once

## Quick Start

### 1. Install Dependencies

```bash
pip install -r requirements.txt
```

### 2. Configure API Key

Copy the template and fill in your API key:

```bash
cp config.json.template config.json
```

Edit `config.json`:

```json
{
  "llm_config": {
    "api_key": "sk-your-api-key-here",
    "api_base": "https://api.openai.com/v1",
    "model": "gpt-4o"
  }
}
```

Any **OpenAI-compatible API** is supported (OpenAI, Azure OpenAI, local LLMs with OpenAI-compatible endpoints, third-party proxies, etc.).

### 3. Add PDF Papers

Place your PDF files in the `papers/` directory.

### 4. Configure Claude Desktop

Edit the Claude Desktop config file:

- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`

Add:

```json
{
  "mcpServers": {
    "paper-extraction": {
      "command": "python",
      "args": ["<full-path-to>/paper-extraction-MCP/server.py"],
      "cwd": "<full-path-to>/paper-extraction-MCP"
    }
  }
}
```

Replace `<full-path-to>` with your actual path. Then restart Claude Desktop.

### 5. Use It

In Claude Desktop, simply say:

```
List the PDF papers available for extraction.
```

```
Extract the paper "my_paper.pdf".
```

```
Extract all papers and show me a summary.
```

## MCP Tools

| Tool | Description |
|---|---|
| `list_papers` | List all PDF files in the `papers/` directory |
| `extract_paper` | Extract metadata + categories from a single PDF |
| `extract_all_papers` | Batch extract all PDFs |
| `get_extraction_result` | Retrieve a previously extracted result (JSON) |

## Project Structure

```
paper-extraction-MCP/
├── server.py              # MCP server (entry point)
├── pdf_extractor.py       # Core extraction logic
├── config.json            # Your configuration (gitignored)
├── config.json.template   # Configuration template
├── requirements.txt       # Python dependencies
├── papers/                # Place PDF files here
├── outputs/
│   ├── json/              # JSON extraction results
│   └── csv/               # CSV extraction results
├── setup.bat              # Windows quick setup
├── setup.sh               # macOS/Linux quick setup
├── CLAUDE_SETUP.md        # Detailed Claude Desktop setup guide
├── LICENSE                # MIT License
└── README.md
```

## How It Works

```
PDF file
  │
  ▼
pdfplumber (text extraction)
  │
  ▼
Full text ──► LLM API call ──► Structured JSON
                  │
          config.json defines:
          - extraction_prompt (fields & rules)
          - llm_config (model, temperature)
                  │
                  ▼
          JSON + CSV output
```

1. **Text Extraction**: `pdfplumber` extracts full text from each PDF page
2. **LLM Extraction**: The text is sent to an LLM with your `extraction_prompt`, which defines what fields to extract and how
3. **Smart Chunking**: If the text exceeds ~100K characters, it is automatically split into chunks, extracted separately, and merged with deduplication
4. **Output**: Results are saved as JSON and CSV

## Built-in Example: Typhoon Disaster Governance

The default `config.json` is pre-configured for extracting information from typhoon disaster governance papers:

**Metadata fields** (5):
- DOI, Title, Journal, Author Affiliations, Publication Date

**Content categories** (7):
| Category | Description |
|---|---|
| Detection & Early Warning | Monitoring, forecasting, alert systems |
| Engineering Protection | Seawalls, drainage, building reinforcement |
| Emergency Response | Evacuation, shelters, rescue operations |
| Post-disaster Recovery | Reconstruction, ecological restoration |
| Policy & Management | Regulations, institutional coordination |
| Digital Technology | AI, big data, remote sensing, GIS, IoT |
| Other Measures | Community-based, education, insurance |

---

## Adapting to Other Domains

The core of this tool is **domain-agnostic**. You only need to modify `config.json` — no code changes required. Here is a step-by-step guide:

### Step 1: Define Your Categories

Decide what information you want to extract. For example:

| Domain | Possible Categories |
|---|---|
| **Climate Change Adaptation** | Mitigation measures, Adaptation strategies, Carbon reduction technologies, Policy instruments, Financial mechanisms |
| **Urban Planning** | Land use strategies, Transportation planning, Green infrastructure, Zoning regulations, Community engagement |
| **Public Health** | Prevention measures, Treatment protocols, Surveillance systems, Policy interventions, Technology applications |
| **Cybersecurity** | Threat detection, Prevention measures, Incident response, Recovery procedures, Governance frameworks |
| **Supply Chain** | Risk identification, Mitigation strategies, Resilience measures, Technology solutions, Regulatory compliance |

### Step 2: Write Your Extraction Prompt

Edit the `extraction_prompt` field in `config.json`. The prompt should:

1. **Describe the assistant's role** for your domain
2. **List metadata fields** (DOI, title, journal, etc. — usually the same across domains)
3. **Define each content category** with clear descriptions and examples
4. **Set extraction rules** (no hallucination, preserve original text, deduplication)
5. **Specify the output JSON format** with exact key names

Here is a **template** you can adapt:

```json
{
  "extraction_prompt": [
    "You are an academic information extraction assistant specialized in [YOUR DOMAIN].",
    "",
    "From each paper, extract:",
    "1) Bibliographic metadata",
    "2) Domain-specific content, categorized as follows:",
    "",
    "METADATA FIELDS:",
    "- 论文DOI: Full DOI URL",
    "- 题目: Paper title",
    "- 期刊名称: Journal name",
    "- 作者机构: Author affiliations (semicolon-separated)",
    "- 发表日期: Publication date (Month Year)",
    "",
    "CONTENT CATEGORIES:",
    "- [Category1_Key]: [Description of what to extract]",
    "- [Category2_Key]: [Description of what to extract]",
    "- ... (add as many as needed)",
    "",
    "RULES:",
    "- Only extract content explicitly present in the paper",
    "- Preserve original text, do not summarize",
    "- Output as JSON with metadata as strings, categories as arrays of strings"
  ]
}
```

### Step 3: Update the Field Mapping in `pdf_extractor.py`

If you change the **Chinese key names** in your extraction prompt (e.g., use `"预防措施"` instead of `"检测预警措施"`), update the `_format_result()` and `_merge_chunk_results()` methods in `pdf_extractor.py` to map your new keys to the internal field names.

For example, if your domain is **public health**:

```python
# In _format_result():
final_result = {
    # ... metadata fields stay the same ...
    "prevention_measures": self._join_measures(result_data.get("预防措施", [])),
    "treatment_measures": self._join_measures(result_data.get("治疗措施", [])),
    "surveillance_measures": self._join_measures(result_data.get("监测措施", [])),
    # ... add your categories ...
}
```

### Step 4: Update `server.py` Display (Optional)

If you want the MCP tool output to show your custom category names, update the `call_tool()` function in `server.py` where it formats the extraction result display.

### Tips for Writing Good Extraction Prompts

1. **Be specific**: Provide concrete examples of what belongs in each category
2. **Set boundaries**: Clearly state what does NOT belong in each category to avoid overlap
3. **Request detail**: Ask for full paragraphs, not just keywords — this prevents information loss
4. **Use the paper's language**: Tell the LLM to preserve the original language (Chinese/English)
5. **Test iteratively**: Try your prompt on 2-3 papers, review the results, and refine

## Configuration Reference

### config.json

| Field | Type | Description |
|---|---|---|
| `papers_dir` | string | Directory containing PDF files (default: `"papers"`) |
| `output_dir` | string | Output directory (default: `"outputs"`) |
| `extraction_prompt` | string or string[] | The LLM prompt defining extraction fields and rules |
| `llm_config.enabled` | bool | Enable/disable LLM extraction |
| `llm_config.provider` | string | LLM provider (currently `"openai"`) |
| `llm_config.model` | string | Model name (e.g., `"gpt-4o"`, `"gpt-4-turbo"`) |
| `llm_config.api_key` | string | Your API key |
| `llm_config.api_base` | string | API base URL |
| `llm_config.temperature` | number | Generation temperature (0 = deterministic) |

### Recommended Models

| Model | Speed | Quality | Cost |
|---|---|---|---|
| `gpt-4o` | Fast | High | Medium |
| `gpt-4-turbo` | Medium | Highest | High |
| `gpt-3.5-turbo` | Fastest | Good | Low |

Any OpenAI-compatible model works (DeepSeek, Qwen, local Ollama, etc.).

## Cost Estimate

Using GPT-4o:
- Single paper (~10 pages): ~$0.04-0.07
- 100 papers: ~$4-7

## Troubleshooting

| Problem | Solution |
|---|---|
| MCP server not visible in Claude | Check config path, restart Claude Desktop |
| API call fails | Verify API key, check network, check account balance |
| Empty extraction | Ensure PDF is text-based (not scanned images) |
| Incomplete results | Paper may be too long — chunking handles this automatically |

See [CLAUDE_SETUP.md](CLAUDE_SETUP.md) for a detailed setup and troubleshooting guide.

## License

[MIT License](LICENSE)

---

If this project helps your research, please give it a star!