Harvester
Harvester
Contents
Harvester is a standalone Go plugin for fetching web and document content,
converting files, searching scholarly and web sources, and exposing the same
tools through MCP. Its source packages live in internal/harvest,
internal/harvestmcp, internal/harvestpy, and internal/harvestcli.
Related MCP server: agentfetch
Commands
The harvester executable supports these entry points:
harvester [flags] <source>...reads URLs, files, and publication identifiers.harvester download-file ...downloads source files without parsing them.harvester ask ...reads the requested sources and asks the configured Claude or Codex CLI to answer.harvester search [--type any|paper|book] ...searches available sources.harvester mcpexposes the Harvester tools over stdio and proxies to the per-userharvester servedaemon.harvester servehosts the persistent Harvester services and optional authenticated external MCP gateway.harvester setup [--offline]provisions conversion support and refreshes the daemon.harvester doctor [--json] [--skip-conversion]checks Harvester's runtime and services.harvester config path|showprints the config path or effective Harvester settings;harvester versionprints the binary version.
The MCP server is named harvester in Claude Code, Codex, and OpenCode. Its
tools are read, search_web, search_literature, and download_file. Every
engine starts the plugin's libexec/harvester mcp launcher over stdio; the
launcher installs a checksum-verified binary on first use, and the proxy
connects to the per-user daemon over its private local endpoint. The daemon
stays available after clients exit.
Engine clients expose those tools under these names:
Claude Code:
mcp__plugin_harvester_harvester__read,mcp__plugin_harvester_harvester__search_web,mcp__plugin_harvester_harvester__search_literature,mcp__plugin_harvester_harvester__download_fileCodex:
mcp__harvester__read,mcp__harvester__search_literature,mcp__harvester__download_file;mcp__harvester__search_webis available when enabled in the Harvester config.OpenCode:
harvester_read,harvester_search_web,harvester_search_literature,harvester_download_file
Read output
read answers one text block per item. Each item shows its content up to
output.maxInlineChars characters (default 50,000); a longer item carries a
truncated: {shown} of {total} chars, full text at the path line, and its
whole artifact sits at the absolute path on the block's second line, ready
to read in pages. include_content: false replaces the content with a
size: {chars} chars line.
Harvester counts and estimates no tokens, and never cuts an answer to fit a
client's output limit: an answer over that limit is the client's to handle.
Claude Code saves an MCP result over its limit (MAX_MCP_OUTPUT_TOKENS,
default 25,000 tokens) to a file and gives the model that file's path in
place of the content. The --json results carry chars, never a token
count, and artifacts carry no token_count frontmatter.
Install and register
Install the native plugin for each engine you use:
Claude Code:
claude plugin marketplace add rezzminator/harvester claude plugin install harvester@professorCodex:
codex plugin marketplace add rezzminator/harvester codex plugin add harvester@professorOpenCode: clone the repository, then install its plugin directory:
git clone https://github.com/rezzminator/harvester cd harvester opencode plugin --global "file://$PWD/plugins/harvester"
The Claude plugin installs from this repository's own marketplace
(.claude-plugin/marketplace.json); the Codex marketplace, named professor, is
.agents/plugins/marketplace.json.
OpenCode loads the packaged plugin's config hook, which registers mcp.harvester
only when no explicit entry already exists. The plugin uses the same launcher
and harvester mcp command on all engines.
Codex registers MCP through plugins/harvester/.codex-plugin/plugin.json,
which points to plugins/harvester/mcp.json. The MCP entry runs the contained
./libexec/harvester mcp command with the installed plugin directory as its
working directory and forwards the configured Harvester, XDG, logging (including PLUGIN_LOG_DIR), and
proxy environment names.
For an engine without the plugin, build or install the standalone executable
and put harvester on PATH. To build from a checkout:
go build -o "$HOME/.local/bin/harvester" ./cmd/harvesterRegister it as a local stdio server with the engine's native command:
Claude Code:
claude mcp add harvester -- harvester mcpCodex:
codex mcp add harvester -- harvester mcpOpenCode: add this entry to
opencode.jsonoropencode.jsonc:{ "mcp": { "harvester": { "type": "local", "command": ["harvester", "mcp"], "enabled": true } } }
These manual registrations invoke the same stdio proxy. Export any Harvester,
XDG, logging, or HTTP proxy settings the daemon should inherit before starting
the engine. For a manual Codex entry in config.toml, list inherited variable
names in env_vars; values come from Codex's environment.
harvester serve is the persistent per-user daemon. It serves local MCP over
the Unix socket ${XDG_STATE_HOME:-$HOME/.local/state}/harvester/daemon.sock
(mode 0600) and owns retrieval, conversion, browser workers, and cache
services. Its singleton lock is
${XDG_STATE_HOME:-$HOME/.local/state}/harvester/daemon.lock (mode 0600);
the daemon's captured output is daemon.log (mode 0600) in the log directory described under Logging. A proxy starts
it detached on demand; it remains running after clients exit. It checks the
installed binary's version once per session and again on a daemon reconnect; it
also rechecks a binary replaced during startup. An older proxy accepts a newer
daemon and never downgrades it, and an older installed build never replaces a
newer running daemon. When a new version or configuration replaces the daemon,
the successor takes the socket at once and serves new calls; the calls the
replaced daemon already took finish on it, for up to 30 minutes.
harvester setup refreshes the daemon after setup to apply current
configuration. harvester status [--json] reports the daemon, and
harvester stop drains and stops it.
When external.enabled is set, the same daemon also serves the optional
authenticated gateway at 127.0.0.1:18378 by default. harvester setup then
installs a Harvester user service (systemd on Linux, LaunchAgent on macOS) so
the gateway is available at login. With the external gateway disabled, setup
installs no service unit; the stdio proxy starts the daemon when needed.
When installed through a native plugin, the service follows the plugin's stable
binary link so plugin version updates take effect without replacing the
service.
Configuration and state
The config path is HARVESTER_CONFIG when set, otherwise
${XDG_CONFIG_HOME:-$HOME/.config}/harvester/harvester.config.json. Harvester
keeps its conversion environment, ask state, cache, and OAuth state under
${XDG_STATE_HOME:-$HOME/.local/state}/harvester/; explicit cache or gateway
state paths in the config remain in use.
Logging
Harvester writes its activity log to $PLUGIN_LOG_DIR/harvester/ when
PLUGIN_LOG_DIR is set, otherwise to ${XDG_STATE_HOME:-$HOME/.local/state}/harvester/logs/;
no Harvester variable or config key moves it, and the daemon's daemon.log
sits beside the level files. The directory is 0700. A record goes to
errors.jsonl, warnings.jsonl, info.jsonl or debug.jsonl (0600) as one
JSON row whose keys are ts, plugin, level, event, msg, where,
session when known, and detail, at most 4096 bytes; the event is the
record's message, msg its err field when it has one, where its component
(else its command), and detail its other scrubbed fields. A level file past
20 MiB moves to {file}.1, older generations shifting up to {file}.5, and a
log-capped row goes to warnings.jsonl. A row that cannot be written is held
for the next write, which writes it first. The screen shows an error as one
line and, once per failing streak, that the log cannot be written and how many
rows are held; nothing else is printed. HARVESTER_LOG_LEVEL and HARVESTER_LOG_COMPONENTS
still choose which records are written. A log left by an earlier version
(harvester.jsonl and its .lock) is not read or removed.
Development
Run the build, vet, and test checks inside the project fence as described in
Testing. plugins/harvester/ contains the native manifests,
OpenCode config hook, and launcher; the launcher verifies release assets
against SHA256SUMS and installs the versioned binary under HARVESTER_HOME.
Available Tools
3 toolsdownload_fileARead-only
Downloads any file as bytes — 1–50, unparsed: a PDF, zip, image, audio, dataset. Call download_file{urls:["https://…/data.zip"]}. Returns a === [n/N] {url} block per item, input order: kind, content type, bytes, sha256, via, then path (local: take the file there) or url and expiry plus a resource_link (remote: run curl -fL -o <file> <url>, check the sha256, never read it into context; a url dies in 10 minutes or at a server restart, so call again). Reading a page or paper → read. A failed item is its header and an error: line; the others still return.
Inputs: urls (required), headers.
readARead-only
Reads pages, files and papers — web pages, local documents and scholarly works, as Markdown. Call read{urls:["https://…"], files:["/path/to/file.pdf"], publications:["10.1038/nature14539"]}: 1–50 items, at most 20 publications. A title goes to search_literature first; a file wanted as bytes to download_file. Returns one text block per item, input order: === [n/N] {item}, the artifact's absolute path, a partial: or truncated: line when incomplete, then the content. A failed item is its header and an error: line; all failed = an error result. Add ask:"question" for one answer over every item, in a last === answer block.
Inputs: ask, urls, files, model, +6 more.
search_literatureARead-only
Finds papers and books — by TITLE or bibliographic query: "find the paper about X", "is there a PDF of ". Call search_literature{query:"Attention Is All You Need", type:"paper"}. Returns a === [n/N] {title} block per ranked candidate — handle, type · authors · year, ids, open access; pass the handle unchanged to read in publications — then sources:, each one's status. "No candidate works found" with every source answered = nothing matched; a failed source is named, its records missing; an error result = every source failed, retry or read an exact identifier.
Inputs: query (required), type, limit.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 3 tools
The three tools map to distinct stages: discover (search_literature), extract as Markdown (read), and fetch raw bytes (download_file). The only real overlap is that both read and download_file can target the same PDF/URL, but the descriptions explicitly split on intent (read for content, download for bytes), so confusion is limited to edge cases.
All names are lowercase snake_case, and search_literature/download_file follow a verb_noun pattern. 'read' is a bare verb rather than verb_noun, a minor deviation but still predictable and readable in context.
Three tools form a tight, well-scoped pipeline (search → read → download) with no redundant entries. Each tool clearly earns its place for a content-harvesting server.
Core lifecycle is covered: discovering scholarly works, reading pages/files/papers, and downloading raw artifacts. The notable gap is discovery of non-scholarly web content — there is no general web search, only literature search — though agents can still read known URLs.
Maintenance
Related MCP Connectors
Shared memory MCP server for Claude Code, Codex, Gemini CLI and Cursor. Cited, editable.
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Document-to-Markdown MCP server — convert PDF, Office and HTML into LLM-ready Markdown.
No-data MCP handoff for local Claude Code to Codex harness moves. $49 lifetime.
Related MCP Servers
- AlicenseAqualityBmaintenanceMCP Server for Web2MD — convert webpage URLs to clean Markdown from Claude Desktop, Cursor, or any MCP-compatible agent.1213 npmMIT
- AlicenseNot gradedqualityDmaintenanceAn open-source web retrieval MCP server that fetches, crawls, and searches the web, returning clean markdown for AI agents. It integrates with Claude MCP, LangChain, and other frameworks for agentic web access.2MIT
- AlicenseAqualityCmaintenanceZero-dependency MCP server that wraps OpenAI Codex's standalone search endpoint into codex_web_search and codex_web_research tools for Claude Code, enabling real-time web search and multi-step deep research independent of the underlying model.23MIT
- AlicenseAqualityBmaintenanceEnables plain-language web search with automatic source and time-window selection, returning relevance-ranked links and summaries to MCP clients such as Claude Code and Codex. Works as an MCP server, CLI, and Claude Code plugin with zero runtime dependencies.17MIT