Harvester
# Harvester
## Contents
- [Commands](#commands)
- [Read output](#read-output)
- [Install and register](#install-and-register)
- [Configuration and state](#configuration-and-state)
- [Development](#development)
Harvester is a standalone Go plugin for fetching web and document content,
converting files, searching scholarly and web sources, and exposing the same
tools through MCP. Its source packages live in `internal/harvest`,
`internal/harvestmcp`, `internal/harvestpy`, and `internal/harvestcli`.
## Commands
The `harvester` executable supports these entry points:
- `harvester [flags] <source>...` reads URLs, files, and publication identifiers.
- `harvester download-file ...` downloads source files without parsing them.
- `harvester ask ...` reads the requested sources and asks the configured
Claude or Codex CLI to answer.
- `harvester search [--type any|paper|book] ...` searches available sources.
- `harvester mcp` exposes the Harvester tools over stdio and proxies to the
per-user `harvester serve` daemon.
- `harvester serve` hosts the persistent Harvester services and optional
authenticated external MCP gateway.
- `harvester setup [--offline]` provisions conversion support and refreshes the
daemon.
- `harvester doctor [--json] [--skip-conversion]` checks Harvester's runtime
and services.
- `harvester config path|show` prints the config path or effective Harvester
settings; `harvester version` prints the binary version.
The MCP server is named `harvester` in Claude Code, Codex, and OpenCode. Its
tools are `read`, `search_web`, `search_literature`, and `download_file`. Every
engine starts the plugin's `libexec/harvester mcp` launcher over stdio; the
launcher installs a checksum-verified binary on first use, and the proxy
connects to the per-user daemon over its private local endpoint. The daemon
stays available after clients exit.
Engine clients expose those tools under these names:
- **Claude Code:** `mcp__plugin_harvester_harvester__read`,
`mcp__plugin_harvester_harvester__search_web`,
`mcp__plugin_harvester_harvester__search_literature`,
`mcp__plugin_harvester_harvester__download_file`
- **Codex:** `mcp__harvester__read`, `mcp__harvester__search_literature`,
`mcp__harvester__download_file`; `mcp__harvester__search_web` is available
when enabled in the Harvester config.
- **OpenCode:** `harvester_read`, `harvester_search_web`,
`harvester_search_literature`, `harvester_download_file`
### Read output
`read` answers one text block per item. Each item shows its content up to
`output.maxInlineChars` characters (default 50,000); a longer item carries a
`truncated: {shown} of {total} chars, full text at the path` line, and its
whole artifact sits at the absolute path on the block's second line, ready
to read in pages. `include_content: false` replaces the content with a
`size: {chars} chars` line.
Harvester counts and estimates no tokens, and never cuts an answer to fit a
client's output limit: an answer over that limit is the client's to handle.
Claude Code saves an MCP result over its limit (`MAX_MCP_OUTPUT_TOKENS`,
default 25,000 tokens) to a file and gives the model that file's path in
place of the content. The `--json` results carry `chars`, never a token
count, and artifacts carry no `token_count` frontmatter.
## Install and register
Install the native plugin for each engine you use:
- **Claude Code:**
```sh
claude plugin marketplace add rezzminator/harvester
claude plugin install harvester@professor
```
- **Codex:**
```sh
codex plugin marketplace add rezzminator/harvester
codex plugin add harvester@professor
```
- **OpenCode:** clone the repository, then install its plugin directory:
```sh
git clone https://github.com/rezzminator/harvester
cd harvester
opencode plugin --global "file://$PWD/plugins/harvester"
```
The Claude plugin installs from this repository's own marketplace
(`.claude-plugin/marketplace.json`); the Codex marketplace, named `professor`, is
`.agents/plugins/marketplace.json`.
OpenCode loads the packaged plugin's config hook, which registers `mcp.harvester`
only when no explicit entry already exists. The plugin uses the same launcher
and `harvester mcp` command on all engines.
Codex registers MCP through `plugins/harvester/.codex-plugin/plugin.json`,
which points to `plugins/harvester/mcp.json`. The MCP entry runs the contained
`./libexec/harvester mcp` command with the installed plugin directory as its
working directory and forwards the configured Harvester, XDG, logging (including `PLUGIN_LOG_DIR`), and
proxy environment names.
For an engine without the plugin, build or install the standalone executable
and put `harvester` on `PATH`. To build from a checkout:
```sh
go build -o "$HOME/.local/bin/harvester" ./cmd/harvester
```
Register it as a local stdio server with the engine's native command:
- **Claude Code:** `claude mcp add harvester -- harvester mcp`
- **Codex:** `codex mcp add harvester -- harvester mcp`
- **OpenCode:** add this entry to `opencode.json` or `opencode.jsonc`:
```json
{
"mcp": {
"harvester": {
"type": "local",
"command": ["harvester", "mcp"],
"enabled": true
}
}
}
```
These manual registrations invoke the same stdio proxy. Export any Harvester,
XDG, logging, or HTTP proxy settings the daemon should inherit before starting
the engine. For a manual Codex entry in `config.toml`, list inherited variable
names in `env_vars`; values come from Codex's environment.
`harvester serve` is the persistent per-user daemon. It serves local MCP over
the Unix socket `${XDG_STATE_HOME:-$HOME/.local/state}/harvester/daemon.sock`
(mode `0600`) and owns retrieval, conversion, browser workers, and cache
services. Its singleton lock is
`${XDG_STATE_HOME:-$HOME/.local/state}/harvester/daemon.lock` (mode `0600`);
the daemon's captured output is `daemon.log` (mode `0600`) in the log directory described under Logging. A proxy starts
it detached on demand; it remains running after clients exit. It checks the
installed binary's version once per session and again on a daemon reconnect; it
also rechecks a binary replaced during startup. An older proxy accepts a newer
daemon and never downgrades it, and an older installed build never replaces a
newer running daemon. When a new version or configuration replaces the daemon,
the successor takes the socket at once and serves new calls; the calls the
replaced daemon already took finish on it, for up to 30 minutes.
`harvester setup` refreshes the daemon after setup to apply current
configuration. `harvester status [--json]` reports the daemon, and
`harvester stop` drains and stops it.
When `external.enabled` is set, the same daemon also serves the optional
authenticated gateway at `127.0.0.1:18378` by default. `harvester setup` then
installs a Harvester user service (systemd on Linux, LaunchAgent on macOS) so
the gateway is available at login. With the external gateway disabled, setup
installs no service unit; the stdio proxy starts the daemon when needed.
When installed through a native plugin, the service follows the plugin's stable
binary link so plugin version updates take effect without replacing the
service.
## Configuration and state
The config path is `HARVESTER_CONFIG` when set, otherwise
`${XDG_CONFIG_HOME:-$HOME/.config}/harvester/harvester.config.json`. Harvester
keeps its conversion environment, ask state, cache, and OAuth state under
`${XDG_STATE_HOME:-$HOME/.local/state}/harvester/`; explicit cache or gateway
state paths in the config remain in use.
### Logging
Harvester writes its activity log to `$PLUGIN_LOG_DIR/harvester/` when
`PLUGIN_LOG_DIR` is set, otherwise to `${XDG_STATE_HOME:-$HOME/.local/state}/harvester/logs/`;
no Harvester variable or config key moves it, and the daemon's `daemon.log`
sits beside the level files. The directory is `0700`. A record goes to
`errors.jsonl`, `warnings.jsonl`, `info.jsonl` or `debug.jsonl` (`0600`) as one
JSON row whose keys are `ts`, `plugin`, `level`, `event`, `msg`, `where`,
`session` when known, and `detail`, at most 4096 bytes; the event is the
record's message, `msg` its `err` field when it has one, `where` its component
(else its command), and `detail` its other scrubbed fields. A level file past
20 MiB moves to `{file}.1`, older generations shifting up to `{file}.5`, and a
`log-capped` row goes to `warnings.jsonl`. A row that cannot be written is held
for the next write, which writes it first. The screen shows an error as one
line and, once per failing streak, that the log cannot be written and how many
rows are held; nothing else is printed. `HARVESTER_LOG_LEVEL` and `HARVESTER_LOG_COMPONENTS`
still choose which records are written. A log left by an earlier version
(`harvester.jsonl` and its `.lock`) is not read or removed.
## Development
Run the build, vet, and test checks inside the project fence as described in
[Testing](docs/testing.md). `plugins/harvester/` contains the native manifests,
OpenCode config hook, and launcher; the launcher verifies release assets
against `SHA256SUMS` and installs the versioned binary under `HARVESTER_HOME`.
TDQS
Scored across 3 tools
The three tools map to distinct stages: discover (search_literature), extract as Markdown (read), and fetch raw bytes (download_file). The only real overlap is that both read and download_file can target the same PDF/URL, but the descriptions explicitly split on intent (read for content, download for bytes), so confusion is limited to edge cases.
All names are lowercase snake_case, and search_literature/download_file follow a verb_noun pattern. 'read' is a bare verb rather than verb_noun, a minor deviation but still predictable and readable in context.
Three tools form a tight, well-scoped pipeline (search → read → download) with no redundant entries. Each tool clearly earns its place for a content-harvesting server.
Core lifecycle is covered: discovering scholarly works, reading pages/files/papers, and downloading raw artifacts. The notable gap is discovery of non-scholarly web content — there is no general web search, only literature search — though agents can still read known URLs.