academic-mcp

A local MCP service that is the whole backend of an academic research workflow: literature search, citation chains, per-project research memory, paper download and PDF→Markdown conversion. Any MCP client can drive it — the first one is the DSH academic agent preset, which contains no Python at all.

client (any MCP host)  ──JSON-RPC /mcp──▶  academic-mcp :8790
                                             ├─ Scopus + OpenAlex    (search, citation chains)
                                             ├─ DeepSeek Search API  (arXiv title rescue)
                                             ├─ project memory JSON  (notes, findings, progress)
                                             ├─ arXiv / Elsevier / browser → PDF
                                             └─ MinerU cloud API     (PDF → Markdown)

Everything that needs Python lives here. A client only sends tool calls.

Tools

convert_document is the one MinerU implementation for both the paper pipeline and general document reading.

Related MCP server: openresearch-mcp

Configuration

Settings come from the process environment, then <MCP_HOME>/.env (this directory), then ~/.shellrc. Copy .env.example to .env and fill it in. Nothing else is read — in particular, no client's directory is consulted.

The essentials:

Full list with defaults and tuning knobs: .env.example. There are two proxy settings and one Elsevier credential: DOWNLOAD_PROXY (publisher/API access, and the rescue route in failover mode) and GFW_PROXY (arXiv/search engines/OpenAlex). ELSEVIER_API_KEY alone is enough for metadata; an institutional insttoken is not supported. DeepSeek's Search API is the exception to both: it is reached directly at api.deepseek.com, the same host the model API uses, so it never goes through GFW_PROXY.

Restart the service after editing (configuration is read once at startup):

systemctl --user restart academic-mcp      # or however you run it

Install

python -m venv .venv && . .venv/bin/activate
pip install -e .                 # or: pip install academic-mcp
cp .env.example .env             # then edit
academic-mcp                     # or: python -m academic_mcp

The browser fallback needs a one-time download and a display (Xvfb is fine):

camoufox fetch

academic-mcp.service and academic-mcp-xvfb.service in this directory are examples — they assume Linux with systemd --user. Adjust the paths before using them.

PDF → Markdown: MinerU only

There is deliberately no fallback converter. The previous backend tried local MinerU (GPU) and then PyMuPDF4LLM; both produced silently degraded text (lost formulas, broken tables, scrambled reading order) that a reading agent then summarised as authoritative. Failing loudly is better than fabricating structure.

Token lifetime and automatic renewal

MINERU_TOKEN is not a permanent API key. It is the OpenXLab SSO token (uaa-token) that mineru.net issues when you log in, and it is a JWT with a hard limited lifetime — 14 days for the token issued on 2026-10-19 (the previous one lasted 90 days, so the number belongs to MinerU, not to this code). When it lapses, every conversion fails with

HTTP 401  {"msgCode": "A0202", "msg": "user authenticate failed"}

while paper downloads keep working — so the symptom is "only the conversion is broken". That is how the outage that began on 2026-10-11 went unnoticed for eight days.

The service now manages this itself. Give it the account you use at https://mineru.net/apiManage (email, phone number or username) and its password:

With both credentials set, the token is renewed:

• at startup, when it is expired or within MINERU_TOKEN_RENEW_DAYS — in a background thread, because a login drives a browser and startup must not wait;

• mid-flight on a 401 — the conversion renews once and retries, so the caller never sees the failure. Parallel conversions share a single login (a stale-token check under a lock), not one browser each.

Manual control, and how to inspect it:

python -m academic_mcp.mineru_auth status           # expiry, source, is renewal configured
python -m academic_mcp.mineru_auth probe            # is the stored token still accepted?
python -m academic_mcp.mineru_auth renew            # log in now
python -m academic_mcp.mineru_auth renew --visible  # if the captcha needs a human

The health tool gained a mineru_token block (expires_at, days_left, expired, needs_renewal, source, renew_enabled), so the state is visible without reading logs.

How it logs in. There is no usable HTTP login: the SSO (sso.openxlab.org.cn) RSA-encrypts the password and demands an Aliyun captcha before it accepts POST /gw/uaa-be/api/v1/login/byAccount — without the captcha it answers A0101 request parameter validation error (measured). Renewal therefore drives the real login page with Camoufox: https://mineru.net/apiManage/token redirects straight to the SSO form, the account and password are filled in, 登录 is clicked, and the 确认您不是机器人 checkbox (#aliyunCaptcha-checkbox-icon) is clicked whenever it appears. The token is then read from the uaa-token cookie and verified against the API before it is stored (a batch id that cannot exist answers "not found" when authenticated and 401 when not — no quota spent).

Where the renewed token goes. Not into .env: the systemd unit runs with ProtectSystem=strict and only ~/.local/state/academic-mcp, the data dir and the cache are writable, so a service-side .env edit would fail with EROFS. The token is written to MINERU_TOKEN_FILE (mode 0600, in a 0700 directory), and whichever of that file and MINERU_TOKEN expires later wins — pasting a fresh token into .env by hand still takes effect immediately, and a stale value in either place can never shadow a newer one.

When renewal fails, it is logged and the conversion fails with the 401 as before; it is never silently degraded to a different converter. Usual causes: a wrong password, an account that requires an SMS/2FA step, or a captcha that escalated to a slider. For the last two, run renew --visible (on the desktop display, where you can drag the slider yourself) and the token is captured and stored as usual.

Resolver order

Cheap and reliable first, expensive and fragile last:

1. arxiv-direct — the DOI is an arXiv ID (10.48550/...)

2. elsevier-api — 10.1016/ + ScienceDirect API key

3. direct-pdf — open-access publishers whose PDF needs no session cookies. Only entries verified to return application/pdf belong here. SciPost is deliberately excluded: it sits behind a proof-of-work gate, and dodging that with a non-browser User-Agent would evade a control someone installed on purpose — it goes through the browser instead.

4. camoufox-browser — headful Firefox, CloudFlare bypass, paywalls. Walks the same routes as the HTTP client (DOWNLOAD_PROXY_MODE), one browser session per exit, because entitlements are per-IP: an exit that is a subscriber for one publisher is a stranger for the next, and the second route is what rescues the first.

5. arxiv-title-search — find the preprint by title (arXiv API first, then a web engine, with every candidate verified)

The last one is a fallback for content, not access: when every publisher path is paywalled, an arXiv preprint of the same paper is still worth reading. It runs last so the version-of-record wins when reachable. Candidates are verified (title similarity + author) before use — fetching the wrong paper is worse than failing. It asks arXiv's own API first and keeps web search as the rescue, because a resolver whose job is to work when the publisher refused us cannot depend on search engines that may be refusing us too.

The rescue engine is DeepSeek's Search API, not DDGS. Measured on 2026-10-04, every DDGS backend was refusing this host (Brave 429, Google 403, DuckDuckGo an empty 202): 23 of 27 title searches failed while the preprints sat on arXiv. DeepSeek answers site:arxiv.org "<title>" (then the leading six words, for titles the phrase match cannot reach) from one endpoint in ~0.5 s, directly — no proxy. It is billed per successful request (5 CNY / 1000, empty results free) and capped at 5 QPS per account; a rescued paper costs at most two requests. Only the result URLs are read: the arXiv ID they carry is a lead, and _verify still checks every candidate against arxiv.org metadata before a byte is downloaded. ARXIV_SEARCH_ENGINE=ddgs restores the old engine, none keeps the API alone; with no DEEPSEEK_API_KEY the default (auto) falls back to DDGS so an existing deployment keeps working.

Display for the browser

Headful Firefox is required for CloudFlare Turnstile (headless is detected and challenged harder). The resolver picks, in order: an existing Xvfb on :99–:96, the ambient DISPLAY, a self-started Xvfb, otherwise headless.

Layout

academic-mcp/
├── academic-mcp.py          # script entrypoint (kept for compatibility)
├── academic_mcp/
│   ├── __main__.py          # python -m academic_mcp
│   ├── config.py            # env + .env settings
│   ├── server.py            # MCP tools (incl. contract / health)
│   ├── pipeline.py          # cache → resolve → convert
│   ├── mineru.py            # MinerU cloud API (the only converter)
│   ├── mineru_auth.py       # token expiry, storage and automatic renewal
│   ├── storage.py           # DOI→key, cache validity
│   ├── httpclient.py        # pooled client, retries, proxy policy
│   ├── websearch.py         # DeepSeek Search API (the arXiv rescue engine)
│   ├── validate.py          # session-scoped DOI authorization
│   ├── markdown.py          # outline + page→line map
│   ├── resolvers/           # arxiv, elsevier, publisher, camoufox
│   └── agent/               # search, snowball, memory + MCP wrappers
├── scripts/                 # manual publisher / network diagnostics
└── tests/                   # offline smoke tests

Development

pip install -e ".[dev]"
ruff check .
pytest -q          # offline: no network, no browser, no MinerU quota

The tests pin the parts clients depend on as a contract: the tool surface, the argument names, the cwd → session_id rule and the memory round-trip.

Security notes

• The service binds to 127.0.0.1 by default and keeps its keys in .env (never returned over the wire; health masks them).

• Automatic MinerU renewal means MINERU_PASSWORD sits in clear text in .env. Keep that file mode 0600 and owned by the service user; the account is the one you use on mineru.net, so give it a unique password. A renewed token is stored separately in MINERU_TOKEN_FILE (also 0600) and is never logged — only a fingerprint (first10…last4) is ever printed.

• validate_doi only authorises a DOI that appeared in this session's search results, so a stray DOI cannot trigger a download.

• The browser fallback drives a real Firefox for CloudFlare challenges and paywalled publishers. Use it in line with those sites' terms.

License

GPL-3.0-or-later — see LICENSE.

Copyright (C) 2026 cxxiao.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. It is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.