lectio
by Kajiih
README.md
# `lectio`
Full-text academic and technical literature reader CLI and stdio Model Context Protocol (**MCP**) server for coding agents.
`lectio` retrieves full primary sources across academic papers (`arXiv`, `DOI`, `PMCID`, `PMID`), formal specifications (`IETF RFCs`, `Python PEPs`, language RFCs), and technical web pages, normalizes them into math-preserving Markdown or confined local PDFs, and returns a compact (`≤ 3,600 B`) provenance header and line- or page-indexed Table of Contents so agents can read exact sections with `read_file` without flooding their context window.
---
## Why `lectio`
- **Structured digital text before binary PDFs**: Keyless open-access (**OA**) waterfalls prefer origin `text/markdown`, `LaTeXML` HTML (`arxiv.org/html`, `ar5iv`), `Europe PMC` `JATS` XML, `RFCXML` v3, and flattened `TeX` source archives ahead of PDFs—preserving verbatim `LaTeX` equations, code listings, and GitHub-Flavored Markdown tables.
- **Adaptive `20`-page PDF workflow**: When only a PDF exists, `lectio` stages the PDF with a companion `<slug>.pages.md` bookmark/page map (pure-Rust `lopdf`, optionally enriched by `pdftotext` when present) and exposes lossless `1`–`8`-page vector PDF slicing (`slice-pdf` / `slice_pdf`) so multimodal agents inspect exact pages without attaching an entire `40`–`150`-page monograph.
- **Epistemic honesty on closed-access works**: Disjoint outcomes (`OK` full text, `OK` PDF, and `METADATA_ONLY`) prevent agents from deriving unverified formulas from abstracts when no legal open-access copy exists.
- **Zero-prompt least-privilege safety**: Exposes read-only `GET`/`HEAD` network retrieval with hop-by-hop IP-pinned SSRF validation, bounded decompression (`25 MiB` HTTP / `40 MiB` TeX archives / `100 MiB` local PDFs), zero-I/O XML DTD-bomb rejection, and `openat(O_DIRECTORY | O_NOFOLLOW)` filesystem confinement blocking configured forbidden prefixes, VCS/credential directories, and agent configuration roots.
---
## Installation
### Standalone Binary (`cargo-binstall` or GitHub Releases)
```bash
cargo binstall lectio
```
Prebuilt archives (`x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`, `aarch64-apple-darwin`, `x86_64-apple-darwin`, `x86_64-pc-windows-msvc`) are attached to each GitHub Release.
### Python `cp310-abi3` Wheel (`uv tool` / Managed macOS)
On managed macOS hosts where binary authorization restricts standalone third-party executables, install the ad-hoc signed Apple Silicon `cp310-abi3` wheel via Homebrew Python and `uv`:
```bash
uv tool install --python /opt/homebrew/opt/python@3.13/bin/python3.13 lectio
```
### From Source
```bash
cargo install --locked --path .
```
---
## Usage
### CLI
```bash
# Fetch an arXiv paper, DOI, PMCID, PMID, IETF RFC, GitHub file, or web URL
lectio fetch 2409.13740
lectio fetch 10.1038/s41586-025-10072-4 -o /tmp/lit
lectio fetch 2409.13740 --no-pdf-index
lectio fetch PMC11456789
lectio fetch "RFC 9110"
lectio fetch https://github.com/rust-lang/rfcs/blob/master/text/2094-nll.md
# Extract a lossless 1..=8-page vector slice from a staged PDF
lectio slice-pdf /tmp/lectio-1000/paper.pdf -p 8-10
lectio slice-pdf /tmp/lectio-1000/paper.pdf -p 8-10 -o /tmp/lit
# Start the newline-delimited JSON-RPC 2.0 stdio MCP server or print version
lectio mcp
lectio --version
```
CLI exit codes: `0` on success (including `BrokenPipe` / `EPIPE` on stdout), `1` on operational, safety, or filesystem errors, and `2` on argument usage errors.
### Stdio MCP Configuration
Register `lectio mcp` with an absolute executable path in your agent's `mcp_config.json`:
```json
{
"mcpServers": {
"lectio": {
"command": "/absolute/path/to/lectio",
"args": ["mcp"]
}
}
}
```
(When bundling inside a plugin manifest that namespaces servers by plugin ID, the server key may be shortened—for example `"mcp"` under a `lectio` plugin.)
The MCP server negotiates protocol versions `2025-06-18`, `2025-03-26`, and `2024-11-05` and exposes two read-only tools:
- `fetch({ target, output_dir?, include_pdf_index? })` (`readOnlyHint: true`, `openWorldHint: true`)
- `slice_pdf({ pdf_path, start_page, end_page?, output_dir? })` (`readOnlyHint: true`, `openWorldHint: false`)
---
## Supported Retrieval Waterfalls
| Target Family | Examples | Resolution Precedence |
| :--- | :--- | :--- |
| **arXiv** | `2409.13740`, `arXiv:2409.13740v1`, `10.48550/arXiv.2409.13740`, `arxiv.org/abs/...` | Atom metadata → `arxiv.org/html` → `ar5iv` (rejects `/abs/` redirects) → `/src` TeX archive → `/pdf` |
| **DOI** | `10.1038/...`, `https://doi.org/...` | `doi.org` CSL-JSON + OpenAlex + optional Unpaywall → discovered PMCID/arXiv → verified OA PDF → substantive non-paywall OA HTML → `METADATA_ONLY` |
| **PMCID / PMID** | `PMC11456789`, `PMID:38123456` | Europe PMC metadata + JATS `fullTextXML` (detects `<front>`-only `BodyRestricted`) → DOI/arXiv fallback → OA PDF → `METADATA_ONLY` |
| **IETF RFC** | `RFC 9110`, `rfc7231` | IETF Datatracker (`obsoleted_by` / `updated_by` notes) → `RFCXML` v3 → form-feed-cleaned `.txt` |
| **Web / GitHub** | `https://...`, `github.com/.../blob/...` | GitHub `/blob/` → raw rewrite; post-redirect scholarly reclassification; `Accept: text/markdown` → HTML5 math extractor / plain text / PDF |
---
## Environment Variables
| Variable | Purpose |
| :--- | :--- |
| `LECTIO_STAGING_DIR` | Optional default staging directory (validated against forbidden prefixes, VCS/config roots, and over-broad `/` or `$HOME`). |
| `LECTIO_EXTRA_ALLOWED_ROOTS` | Optional OS path-list (`:` on Unix, `;` on Windows) of additional allowed staging roots (e.g., agent scratch directories). |
| `LECTIO_FORBIDDEN_PATH_PREFIXES` | Optional OS path-list of additional absolute path prefixes where staging is blocked fail-closed. |
| `LECTIO_BLOCKED_HOST_SUFFIXES` | Optional comma-separated host suffixes blocked from outbound HTTP(S) retrieval (exact domain and subdomains). |
| `LECTIO_CONTACT_EMAIL` | Optional non-placeholder email enabling Unpaywall v2 and OpenAlex polite-pool `mailto:` queries. |
| `LECTIO_DISABLE_PDF_SLICE` | Truthy value (`1`, `true`, `yes`, `on`) disables local PDF slicing (`SafetyBlock`). |
| `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, `NO_PROXY` | Standard uppercase/lowercase HTTP(S) proxy configuration (SOCKS proxies are rejected explicitly). |
---
## Architecture & Development
`lectio` is structured around a compile-time-verified component DAG in [src/architecture.rs](src/architecture.rs) checked by [tests/architecture_conformance.rs](tests/architecture_conformance.rs):
```bash
cargo fmt --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo doc --workspace --no-deps
omnicoach code
```
## References
- [PaperQA2 (Skarlinski et al., 2024)](https://arxiv.org/abs/2409.13740) — Full-text staging and contextual summarization ahead of synthesis.
- [OpenScholar (Asai et al., Nature 2026)](https://doi.org/10.1038/s41586-025-10072-4) — Full-text passage retrieval and citation verification.
- [ReadAgent (Lee et al., 2024)](https://arxiv.org/abs/2402.09727) — Compact gist Table of Contents paired with verbatim section lookup.
- [Model Context Protocol Specification (`2025-06-18`)](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) — Newline-delimited JSON-RPC `2.0` stdio transport and batch-array removal.
- [ISO 32000-1:2008 PDF 1.7 Specification, §7.7.3.4](https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf) — Inheritable `/Pages` attributes materialized during PDF slicing.
## License
MIT — see [LICENSE](LICENSE).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues