Skip to main content
Glama

Harvester

Contents

Harvester is a standalone Go plugin for fetching web and document content, converting files, searching scholarly and web sources, and exposing the same tools through MCP. Its source packages live in internal/harvest, internal/harvestmcp, internal/harvestpy, and internal/harvestcli.

Related MCP server: agentfetch

Commands

The harvester executable supports these entry points:

  • harvester [flags] <source>... reads URLs, files, and publication identifiers.

  • harvester download-file ... downloads source files without parsing them.

  • harvester ask ... reads the requested sources and asks the configured Claude or Codex CLI to answer.

  • harvester search [--type any|paper|book] ... searches available sources.

  • harvester mcp exposes the Harvester tools over stdio and proxies to the per-user harvester serve daemon.

  • harvester serve hosts the persistent Harvester services and optional authenticated external MCP gateway.

  • harvester setup [--offline] provisions conversion support and refreshes the daemon.

  • harvester doctor [--json] [--skip-conversion] checks Harvester's runtime and services.

  • harvester config path|show prints the config path or effective Harvester settings; harvester version prints the binary version.

The MCP server is named harvester in Claude Code, Codex, and OpenCode. Its tools are read, search_web, search_literature, and download_file. Every engine starts the plugin's libexec/harvester mcp launcher over stdio; the launcher installs a checksum-verified binary on first use, and the proxy connects to the per-user daemon over its private local endpoint. The daemon stays available after clients exit.

Engine clients expose those tools under these names:

  • Claude Code: mcp__plugin_harvester_harvester__read, mcp__plugin_harvester_harvester__search_web, mcp__plugin_harvester_harvester__search_literature, mcp__plugin_harvester_harvester__download_file

  • Codex: mcp__harvester__read, mcp__harvester__search_literature, mcp__harvester__download_file; mcp__harvester__search_web is available when enabled in the Harvester config.

  • OpenCode: harvester_read, harvester_search_web, harvester_search_literature, harvester_download_file

Read output

read answers one text block per item. Each item shows its content up to output.maxInlineChars characters (default 50,000); a longer item carries a truncated: {shown} of {total} chars, full text at the path line, and its whole artifact sits at the absolute path on the block's second line, ready to read in pages. include_content: false replaces the content with a size: {chars} chars line.

Harvester counts and estimates no tokens, and never cuts an answer to fit a client's output limit: an answer over that limit is the client's to handle. Claude Code saves an MCP result over its limit (MAX_MCP_OUTPUT_TOKENS, default 25,000 tokens) to a file and gives the model that file's path in place of the content. The --json results carry chars, never a token count, and artifacts carry no token_count frontmatter.

Install and register

Install the native plugin for each engine you use:

  • Claude Code:

    claude plugin marketplace add rezzminator/harvester
    claude plugin install harvester@professor
  • Codex:

    codex plugin marketplace add rezzminator/harvester
    codex plugin add harvester@professor
  • OpenCode: clone the repository, then install its plugin directory:

    git clone https://github.com/rezzminator/harvester
    cd harvester
    opencode plugin --global "file://$PWD/plugins/harvester"

The Claude plugin installs from this repository's own marketplace (.claude-plugin/marketplace.json); the Codex marketplace, named professor, is .agents/plugins/marketplace.json. OpenCode loads the packaged plugin's config hook, which registers mcp.harvester only when no explicit entry already exists. The plugin uses the same launcher and harvester mcp command on all engines.

Codex registers MCP through plugins/harvester/.codex-plugin/plugin.json, which points to plugins/harvester/mcp.json. The MCP entry runs the contained ./libexec/harvester mcp command with the installed plugin directory as its working directory and forwards the configured Harvester, XDG, logging (including PLUGIN_LOG_DIR), and proxy environment names.

For an engine without the plugin, build or install the standalone executable and put harvester on PATH. To build from a checkout:

go build -o "$HOME/.local/bin/harvester" ./cmd/harvester

Register it as a local stdio server with the engine's native command:

  • Claude Code: claude mcp add harvester -- harvester mcp

  • Codex: codex mcp add harvester -- harvester mcp

  • OpenCode: add this entry to opencode.json or opencode.jsonc:

    {
      "mcp": {
        "harvester": {
          "type": "local",
          "command": ["harvester", "mcp"],
          "enabled": true
        }
      }
    }

These manual registrations invoke the same stdio proxy. Export any Harvester, XDG, logging, or HTTP proxy settings the daemon should inherit before starting the engine. For a manual Codex entry in config.toml, list inherited variable names in env_vars; values come from Codex's environment.

harvester serve is the persistent per-user daemon. It serves local MCP over the Unix socket ${XDG_STATE_HOME:-$HOME/.local/state}/harvester/daemon.sock (mode 0600) and owns retrieval, conversion, browser workers, and cache services. Its singleton lock is ${XDG_STATE_HOME:-$HOME/.local/state}/harvester/daemon.lock (mode 0600); the daemon's captured output is daemon.log (mode 0600) in the log directory described under Logging. A proxy starts it detached on demand; it remains running after clients exit. It checks the installed binary's version once per session and again on a daemon reconnect; it also rechecks a binary replaced during startup. An older proxy accepts a newer daemon and never downgrades it, and an older installed build never replaces a newer running daemon. When a new version or configuration replaces the daemon, the successor takes the socket at once and serves new calls; the calls the replaced daemon already took finish on it, for up to 30 minutes. harvester setup refreshes the daemon after setup to apply current configuration. harvester status [--json] reports the daemon, and harvester stop drains and stops it.

When external.enabled is set, the same daemon also serves the optional authenticated gateway at 127.0.0.1:18378 by default. harvester setup then installs a Harvester user service (systemd on Linux, LaunchAgent on macOS) so the gateway is available at login. With the external gateway disabled, setup installs no service unit; the stdio proxy starts the daemon when needed. When installed through a native plugin, the service follows the plugin's stable binary link so plugin version updates take effect without replacing the service.

Configuration and state

The config path is HARVESTER_CONFIG when set, otherwise ${XDG_CONFIG_HOME:-$HOME/.config}/harvester/harvester.config.json. Harvester keeps its conversion environment, ask state, cache, and OAuth state under ${XDG_STATE_HOME:-$HOME/.local/state}/harvester/; explicit cache or gateway state paths in the config remain in use.

Logging

Harvester writes its activity log to $PLUGIN_LOG_DIR/harvester/ when PLUGIN_LOG_DIR is set, otherwise to ${XDG_STATE_HOME:-$HOME/.local/state}/harvester/logs/; no Harvester variable or config key moves it, and the daemon's daemon.log sits beside the level files. The directory is 0700. A record goes to errors.jsonl, warnings.jsonl, info.jsonl or debug.jsonl (0600) as one JSON row whose keys are ts, plugin, level, event, msg, where, session when known, and detail, at most 4096 bytes; the event is the record's message, msg its err field when it has one, where its component (else its command), and detail its other scrubbed fields. A level file past 20 MiB moves to {file}.1, older generations shifting up to {file}.5, and a log-capped row goes to warnings.jsonl. A row that cannot be written is held for the next write, which writes it first. The screen shows an error as one line and, once per failing streak, that the log cannot be written and how many rows are held; nothing else is printed. HARVESTER_LOG_LEVEL and HARVESTER_LOG_COMPONENTS still choose which records are written. A log left by an earlier version (harvester.jsonl and its .lock) is not read or removed.

Development

Run the build, vet, and test checks inside the project fence as described in Testing. plugins/harvester/ contains the native manifests, OpenCode config hook, and launcher; the launcher verifies release assets against SHA256SUMS and installs the versioned binary under HARVESTER_HOME.

Available Tools

3 tools
download_fileA
Read-only

Downloads any file as bytes — 1–50, unparsed: a PDF, zip, image, audio, dataset. Call download_file{urls:["https://…/data.zip"]}. Returns a === [n/N] {url} block per item, input order: kind, content type, bytes, sha256, via, then path (local: take the file there) or url and expiry plus a resource_link (remote: run curl -fL -o <file> <url>, check the sha256, never read it into context; a url dies in 10 minutes or at a server restart, so call again). Reading a page or paper → read. A failed item is its header and an error: line; the others still return.

Inputs: urls (required), headers.

readA
Read-only

Reads pages, files and papers — web pages, local documents and scholarly works, as Markdown. Call read{urls:["https://…"], files:["/path/to/file.pdf"], publications:["10.1038/nature14539"]}: 1–50 items, at most 20 publications. A title goes to search_literature first; a file wanted as bytes to download_file. Returns one text block per item, input order: === [n/N] {item}, the artifact's absolute path, a partial: or truncated: line when incomplete, then the content. A failed item is its header and an error: line; all failed = an error result. Add ask:"question" for one answer over every item, in a last === answer block.

Inputs: ask, urls, files, model, +6 more.

search_literatureA
Read-only

Finds papers and books — by TITLE or bibliographic query: "find the paper about X", "is there a PDF of ". Call search_literature{query:"Attention Is All You Need", type:"paper"}. Returns a === [n/N] {title} block per ranked candidate — handle, type · authors · year, ids, open access; pass the handle unchanged to read in publications — then sources:, each one's status. "No candidate works found" with every source answered = nothing matched; a failed source is named, its records missing; an error result = every source failed, retry or read an exact identifier.

Inputs: query (required), type, limit.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.4/5.0

    Scored across 3 tools

    Disambiguation4/5

    The three tools map to distinct stages: discover (search_literature), extract as Markdown (read), and fetch raw bytes (download_file). The only real overlap is that both read and download_file can target the same PDF/URL, but the descriptions explicitly split on intent (read for content, download for bytes), so confusion is limited to edge cases.

    Naming Consistency4/5

    All names are lowercase snake_case, and search_literature/download_file follow a verb_noun pattern. 'read' is a bare verb rather than verb_noun, a minor deviation but still predictable and readable in context.

    Tool Count5/5

    Three tools form a tight, well-scoped pipeline (search → read → download) with no redundant entries. Each tool clearly earns its place for a content-harvesting server.

    Completeness4/5

    Core lifecycle is covered: discovering scholarly works, reading pages/files/papers, and downloading raw artifacts. The notable gap is discovery of non-scholarly web content — there is no general web search, only literature search — though agents can still read known URLs.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      An open-source web retrieval MCP server that fetches, crawls, and searches the web, returning clean markdown for AI agents. It integrates with Claude MCP, LangChain, and other frameworks for agentic web access.
      2
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Zero-dependency MCP server that wraps OpenAI Codex's standalone search endpoint into codex_web_search and codex_web_research tools for Claude Code, enabling real-time web search and multi-step deep research independent of the underlying model.
      2
      3
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables plain-language web search with automatic source and time-window selection, returning relevance-ranked links and summaries to MCP clients such as Claude Code and Codex. Works as an MCP server, CLI, and Claude Code plugin with zero runtime dependencies.
      1
      7
      MIT