arXiv MCP Server

# arXiv CLI & MCP Server A Python toolkit for searching and downloading papers from arXiv.org, with both a command-line interface and a Model Context Protocol (MCP) server for LLM integration. CLI agents work well with well-documented CLI tools and/or MCP servers. This project provides both options. ## Features - **Search** arXiv papers by title, author, abstract, category, and more - **Download** PDFs automatically with local caching - **MCP Server** for integration with LLM assistants (Claude Desktop, etc.) - **Typed responses** using Pydantic models for clean data handling - **Rate limiting** built-in to respect arXiv API guidelines - **Comprehensive tests** with 26 integration tests (no mocking) ## Installation ### Option 1: Install from GitHub (Recommended) Install directly from the GitHub repository: ```bash # Install the latest version uv pip install git+https://github.com/LiamConnell/arxiv_for_agents.git # Or with pip pip install git+https://github.com/LiamConnell/arxiv_for_agents.git # Now you can use the arxiv command arxiv --help ``` ### Option 2: Install from Source Clone the repository and install locally: ```bash # Clone the repository git clone https://github.com/LiamConnell/arxiv_for_agents.git cd arxiv_for_agents # Install in editable mode uv pip install -e . # Now you can use the arxiv command arxiv --help ``` ### Option 3: Development Installation For development with all dependencies: ```bash # Clone and install with dev dependencies git clone https://github.com/LiamConnell/arxiv_for_agents.git cd arxiv_for_agents uv pip install -e ".[dev]" # Run tests uv run pytest ``` ### Verify Installation ```bash # If installed as package arxiv --help # Or if using as module uv run python -m arxiv --help ``` ## Usage **Note:** If you installed as a package, use `arxiv` directly. Otherwise, use `uv run python -m arxiv`. ### Search Papers Search by title: ```bash # Using installed package arxiv search "ti:attention is all you need" # Or using as module uv run python -m arxiv search "ti:attention is all you need" ``` Search by author: ```bash arxiv search "au:Hinton" --max-results 20 ``` Search by category: ```bash arxiv search "cat:cs.AI" --max-results 10 ``` Combined search: ```bash arxiv search "ti:transformer AND au:Vaswani" ``` ### Get Specific Paper Get paper metadata and download PDF: ```bash arxiv get 1706.03762 ``` Get metadata only (no download): ```bash arxiv get 1706.03762 --no-download ``` Force re-download: ```bash arxiv get 1706.03762 --force ``` ### Download PDF Download just the PDF: ```bash arxiv download 1706.03762 ``` ### List Downloaded PDFs ```bash arxiv list-downloads ``` ### JSON Output Get results as JSON for scripting: ```bash arxiv search "ti:neural" --json arxiv get 1706.03762 --json --no-download ``` ## Search Query Syntax The arXiv API supports field-specific searches: - `ti:` - Title - `au:` - Author - `abs:` - Abstract - `cat:` - Category (e.g., cs.AI, cs.LG) - `all:` - All fields (default) You can combine searches with `AND`, `OR`, and `ANDNOT`: ```bash arxiv search "ti:neural AND cat:cs.LG" arxiv search "au:Hinton OR au:Bengio" ``` ## Download Directory PDFs are downloaded to `./.arxiv` by default. Change this with: ```bash arxiv --download-dir ./papers search "ti:transformer" ``` ## MCP Server (Model Context Protocol) The arXiv CLI includes a Model Context Protocol (MCP) server that allows LLM assistants (like Claude Desktop) to search and download arXiv papers programmatically. ### Running the MCP Server ```bash # Option 1: Using the script entry point (recommended) uv run arxiv-mcp # Option 2: Using the module uv run python -m arxiv.mcp ``` The server runs in stdio mode and communicates via JSON-RPC over stdin/stdout. ### MCP Tools The server provides 4 tools for paper discovery and management: 1. **search_papers** - Search arXiv with advanced query syntax - Supports field prefixes (ti:, au:, abs:, cat:) - Boolean operators (AND, OR, ANDNOT) - Pagination and sorting options - Returns paper metadata including title, authors, abstract, categories 2. **get_paper** - Get detailed information about a specific paper - Accepts flexible ID formats (1706.03762, arXiv:1706.03762, 1706.03762v1) - Optionally downloads PDF automatically - Returns complete metadata including DOI, journal references, comments 3. **download_paper** - Download PDF for a specific paper - Downloads to local `.arxiv` directory - Returns file path and size information - Supports force re-download option 4. **list_downloaded_papers** - List all locally downloaded PDFs - Shows arxiv IDs, file sizes, and paths - Useful for managing local paper collection ### MCP Resources The server exposes 2 resources for direct access: - **paper://{arxiv_id}** - Get formatted paper metadata in markdown - **downloads://list** - Get markdown table of all downloaded papers ### MCP Prompts Pre-built prompt templates to guide usage: - **search_arxiv_prompt** - Guide for searching arXiv papers - **download_paper_prompt** - Guide for downloading and managing papers ### Claude Desktop Configuration Add to your Claude Desktop config file (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS): **If installed from GitHub/pip:** ```json { "mcpServers": { "arxiv": { "command": "arxiv-mcp" } } } ``` **If running from source/development:** ```json { "mcpServers": { "arxiv": { "command": "uv", "args": ["run", "arxiv-mcp"], "cwd": "/path/to/arxiv_for_agents" } } } ``` Or use `--directory` to avoid needing `cwd`: ```json { "mcpServers": { "arxiv": { "command": "uv", "args": ["--directory", "/path/to/arxiv_for_agents", "run", "arxiv-mcp"] } } } ``` ### MCP Use Cases Once configured, you can ask Claude to: - "Search arXiv for recent papers on transformer architectures" - "Find papers by Geoffrey Hinton in the cs.AI category" - "Download the 'Attention is All You Need' paper" - "Show me papers about neural networks from 2023" - "List all the papers I've downloaded" - "Get the abstract for arXiv:1706.03762" The MCP integration allows Claude to autonomously search, retrieve, and manage academic papers from arXiv. ## Architecture ### Module Structure ``` arxiv/ ├── __init__.py # Package exports ├── __main__.py # CLI entry point ├── cli.py # Click commands ├── models.py # Pydantic models ├── services.py # API client service └── mcp/ # MCP server ├── __init__.py # MCP package exports ├── __main__.py # MCP server entry point └── server.py # FastMCP server with tools, resources, prompts tests/ └── test_services.py # Integration tests (26 tests) ``` ### Pydantic Models All API responses are typed using Pydantic: ```python from arxiv import ArxivService service = ArxivService() result = service.search("ti:neural", max_results=5) # result is typed as ArxivSearchResult print(f"Total: {result.total_results}") for entry in result.entries: # entry is typed as ArxivEntry print(f"{entry.arxiv_id}: {entry.title}") print(f"Authors: {', '.join(a.name for a in entry.authors)}") ``` ### Key Models - **ArxivSearchResult**: Search results with metadata - `total_results`: Total matching papers - `entries`: List of ArxivEntry objects - **ArxivEntry**: Individual paper - `arxiv_id`: Clean ID (e.g., "1706.03762") - `title`, `summary`: Paper metadata - `authors`: List of Author objects - `categories`: Subject categories - `pdf_url`: Direct PDF link - `published`, `updated`: Datetime objects - **Author**: Paper author - `name`: Author name - `affiliation`: Optional affiliation ## Testing Run all 26 integration tests (makes real API calls): ```bash uv run pytest tests/test_services.py -v ``` Run specific test class: ```bash uv run pytest tests/test_services.py::TestArxivServiceSearch -v ``` The tests are integration tests that hit the real arXiv API, ensuring the service works with actual data. ## API Rate Limiting The service enforces a 3-second delay between API requests by default (arXiv's recommendation). You can adjust this: ```python from arxiv import ArxivService service = ArxivService(rate_limit_delay=5.0) # 5 seconds ``` ## Examples ### Python API ```python from arxiv import ArxivService # Initialize service service = ArxivService(download_dir="./papers") # Search results = service.search( query="ti:attention is all you need", max_results=5, sort_by="relevance" ) print(f"Found {results.total_results} papers") for entry in results.entries: print(f"- {entry.title}") # Get specific paper entry = service.get("1706.03762", download_pdf=True) print(f"Downloaded: {entry.title}") # Just download PDF pdf_path = service.download_pdf("1706.03762") print(f"PDF saved to: {pdf_path}") ``` ### CLI Examples ```bash # Find recent papers in a category arxiv search "cat:cs.AI" \ --max-results 10 \ --sort-by submittedDate \ --sort-order descending # Search and output as JSON for processing arxiv search "ti:transformer" --json | jq '.entries[].title' # Batch download multiple papers for id in 1706.03762 1810.04805 2010.11929; do arxiv download $id done ``` ## Development The codebase follows these principles: 1. **Type safety**: Pydantic models for all API responses 2. **Clean architecture**: Separation of CLI, service, and models 3. **Real tests**: Integration tests with actual API calls (no mocks) 4. **Rate limiting**: Respects arXiv API guidelines 5. **Caching**: Automatic local caching to avoid re-downloads ## arXiv API Reference - Base URL: https://export.arxiv.org/api/query - Format: Atom XML - Rate limit: 3 seconds between requests (recommended) - Documentation: https://info.arxiv.org/help/api/user-manual.html ## License This is a personal project for interacting with arXiv's public API.