Searches GitHub repositories and integrates them as training-data sources, letting agents discover code, datasets, and related repos for a topic.
Searches Hugging Face datasets and integrates the results into training-data packs, letting agents discover and select relevant datasets for a given topic.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@harvest find me training data for a small vision-language model"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Plain-English topic in → 12 experts fan out over Hugging Face + GitHub in parallel → coverage-scored training pack JSON out in ~30 seconds.
No API keys. No Bridge. No account. It works out of the box.
Harvest is a training-data scout: you describe what you want to train on in plain English
("image classification", "medical question answering"), and 12 domain experts —
code, math, science, language, vision, audio, medical, law, knowledge, safety, affect, systems —
each sweep Hugging Face datasets and GitHub repos with their own curated queries. The local mixer
then composes the hits into a balanced, coverage-scored training library pack
(flare-harvest-library v1 JSON) you can hand to any training pipeline.
npx github:The-Warl0ck/harvest-mcpOr add it to Claude Desktop (claude_desktop_config.json):
{
"mcpServers": {
"harvest": {
"command": "npx",
"args": ["github:The-Warl0ck/harvest-mcp"]
}
}
}With optional tokens for higher rate limits:
{
"mcpServers": {
"harvest": {
"command": "npx",
"args": ["github:The-Warl0ck/harvest-mcp"],
"env": {
"HF_TOKEN": "hf_...",
"GITHUB_TOKEN": "ghp_..."
}
}
}
}Requirements: Node.js ≥ 20.
Related MCP server: GitHub Assistant MCP Server
You: find me training data for a small vision-language model
harvest.search
{ "topic": "vision language instruction tuning" }→ ~40 hits: LLaVA-NeXT data, the Cauldron, DOCCI, LLaVA repo… plus the 12-expert atlas sweep for breadth (code, math, safety…)
harvest.compose
{ "goal": "small vision-language model", "catalog": […] }→{ "title": "small vision-language model", "picks": [ { "id": "lmms-lab/LLaVA-NeXT-Data", "expert": "vision", "why": "Anchor Vision on Hugging Face" }, … ], "rationale": "Local mixer staged 22 items across 12/12 experts. No mouth required.", "engine": "local" }
harvest.pack
{ "goal": "small vision-language model", "name": "VLM Mix v1", "catalog": […] }→flare-harvest-libraryv1 JSON withcoverage: { filled: 12, totalExperts: 12, balance: 0.94 }and per-itemingest: { type: "hf-dataset" | "github-repo", ref }pointers.
Tool | What it does |
|
|
|
|
|
|
|
|
|
|
catalog items are { id, source, expert, description? } — the objects harvest.search returns drop straight in.
Everything hits the public Hugging Face and GitHub APIs. Unauthenticated, you get:
Hugging Face: generous anonymous quota on /api/datasets (occasional 429s on bursts; results are cached 12 min in-process)
GitHub: 60 requests/hour per IP for search — the 12-expert sweep is the hungriest call
For real use, set HF_TOKEN / GITHUB_TOKEN (free accounts) via env or the per-call
hf_token / gh_token args. See .env.example.
Packs are flare-harvest-library v1 JSON — see PACK-FORMAT.md.
src/harvest/scan-core.ts — parallel HF/GitHub scanning, 12-min result cache
src/experts.ts — the 12-expert atlas (curated queries + seed items)
src/harvest/compose-local.ts — the local mixer ("No mouth required"): anchors one item per expert, then balances HF/GitHub sources, then fills for coverage
src/coverage.ts — entropy-based coverage/balance scoring across the 12 experts
src/harvest/compose-mouth.ts — opt-in: compose via any OpenAI-compatible /v1 endpoint (LM Studio, etc.) instead of the local mixer
mcp/server.ts — the MCP server (stdio, low-level SDK API, hand-written schemas)
Apache-2.0 — see LICENSE.
Compose a balanced mix from a catalog of items with the local mixer — no LLM/mouth required. Returns picks with rationale.
| Name | Required | Description | Default |
|---|---|---|---|
| goal | Yes | What the training library is for, e.g. 'small vision-language model'. | |
| catalog | Yes | Candidate items (e.g. from harvest.search). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. It adds useful context ('local mixer — no LLM/mouth required' and 'returns picks with rationale'), but it does not discuss permissions, side effects, failure modes, or how 'balanced' is determined. The unusual 'no LLM/mouth' phrasing also reduces clarity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core action before the return statement. The only blemish is the unclear 'no LLM/mouth' phrase, but the sentence is otherwise economical and free of redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-required-param tool with a fully described schema and an output statement, this is workable. However, it never defines what 'balanced' means, when composition might fail, or how the rationale is structured, and there is no output schema to fill those gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters. The plain-language description adds context by framing the catalog as candidate items and the goal as the purpose of the training library, but it does not substantially enrich parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a concrete action ('compose a balanced mix'), a specific resource ('a catalog of items'), and an output ('returns picks with rationale'). It distinguishes the tool from search-like siblings by emphasizing the local mixer and selection output, though it does not explicitly name alternatives like harvest.pack.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied: supply a goal and a catalog of candidate items to get a balanced selection. There is no explicit when-to-use guidance, no mention of when to prefer a sibling, and no exclusion criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run one of the 12 experts against Hugging Face + GitHub using its curated queries. Returns HarvestItem[].
| Name | Required | Description | Default |
|---|---|---|---|
| expert | Yes | Expert id: code, math, science, language, vision, audio, medical, law, knowledge, safety, affect, systems. | |
| gh_token | No | Optional GitHub token for higher rate limits. Falls back to the GITHUB_TOKEN env var. | |
| hf_token | No | Optional Hugging Face token for higher rate limits. Falls back to the HF_TOKEN env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that the tool runs a query and returns HarvestItem[], which is useful, but it does not mention whether the operation is read-only, potential rate-limit behavior, or any side effects. The token parameters hint at authentication but are not described in the description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, dense sentence that front-loads the core action and includes the return type. There is no filler or redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description conveys the essential operation, data sources, and return type, which is helpful given the absence of an output schema. However, it omits usage guidance relative to siblings and behavioral details such as rate limits or read-only guarantees, leaving some gaps for an agent deciding when to invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds no additional parameter meaning beyond the schema, but the schema already documents expert, gh_token, and hf_token adequately.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Run'), the resource ('one of the 12 experts'), and the targets ('Hugging Face + GitHub'). It is specific enough to understand the tool's role, though it does not explicitly distinguish itself from sibling tools like harvest.search.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'using its curated queries' implies this tool is for expert-driven searches rather than general search, but it does not explicitly state when to prefer this over harvest.search or the other siblings. Usage context is implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scan what's trending: recently-updated Hugging Face datasets and hot/recent GitHub repos. Returns HarvestItem[].
| Name | Required | Description | Default |
|---|---|---|---|
| gh_token | No | Optional GitHub token for higher rate limits. Falls back to the GITHUB_TOKEN env var. | |
| hf_token | No | Optional Hugging Face token for higher rate limits. Falls back to the HF_TOKEN env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure, but it only states purpose and return type. It does not reveal ordering, result limits, pagination, or the operational meaning of 'hot/recent'. The rate-limit fallback behavior is documented only in the schema parameter descriptions, not in the tool description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact sentences with zero filler: the action and scope are front-loaded, and the return-type note earns its place given the absence of an output schema. Nothing redundant remains.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, has zero required parameters, and the description covers the core purpose, making it callable with minimal confusion. Yet the description only names HarvestItem[] without defining its structure, and it omits pagination or ordering behavior, leaving some ambiguity about the result contents and size.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and each parameter already carries clear semantics: optional GitHub/Hugging Face tokens with env-var fallback for higher rate limits. The tool description adds no parameter-specific information beyond what the schema provides, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Scan') and identifies concrete resources: recently-updated Hugging Face datasets and hot/recent GitHub repos. It also names the return type HarvestItem[], and the 'trending'/'latest' framing clearly distinguishes it from the query-oriented sibling harvest.search.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the intended use case: when the agent wants trending or recent items rather than targeted searches. However, it never explicitly names alternatives, states when not to use this tool, or explains how it relates to harvest.search, harvest.expert, harvest.compose, or harvest.pack.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compose a mix from a catalog and emit a flare-harvest-library v1 pack JSON with coverage scores. No Bridge stanza included.
| Name | Required | Description | Default |
|---|---|---|---|
| goal | Yes | What the training library is for. | |
| name | Yes | Pack name, e.g. 'Vision Mix v1'. | |
| catalog | Yes | Candidate items (e.g. from harvest.search). Full HarvestItem objects preferred. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses the output format (flare-harvest-library v1 pack JSON with coverage scores) and the absence of a Bridge stanza, which is useful. However, it does not disclose side effects (e.g., whether anything is persisted), performance characteristics, or failure modes. The description is honest but incomplete for a tool that likely performs computation and returns a structured artifact.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no wasted words. The core action and output format are front-loaded, and the Bridge stanza exclusion is a useful differentiator. It could be slightly more structured by naming the sibling it is not, but it is appropriately sized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 3 required parameters, 100% schema coverage, and no output schema, the description is adequate but not complete. It tells the agent what the tool produces and what it omits, but it does not explain how the 'goal' parameter affects the mix, what 'coverage scores' mean, or how this relates to harvest.compose. An agent could invoke it correctly from the schema, but might not know when to prefer it over a sibling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all three parameters. The description adds minimal parameter-level meaning: 'goal' is described as 'what the training library is for' in the schema, and the description's 'Compose a mix from a catalog' reinforces the role of 'catalog'. It does not add details about how 'name' is used or how 'goal' influences composition. Baseline 3 is appropriate because the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Compose a mix from a catalog') and a concrete deliverable ('emit a flare-harvest-library v1 pack JSON with coverage scores'). It also notes the absence of a Bridge stanza, which distinguishes it from a potential sibling. However, it does not explicitly name a sibling tool or contrast with harvest.compose, so differentiation is implied rather than explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage: given a catalog of candidate items, compose a mix and emit a pack JSON. It does not state when to use this tool versus harvest.compose or harvest.search, nor does it provide exclusions or alternative routing. The sibling list includes harvest.compose, which likely overlaps, but the description does not clarify the boundary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search Hugging Face datasets and GitHub repos for a plain-English topic, then sweep the 12-expert atlas for breadth. Returns deduplicated HarvestItem[]. Works with no tokens (public rate limits apply).
| Name | Required | Description | Default |
|---|---|---|---|
| topic | Yes | Plain-English topic, e.g. 'image classification' or 'speech recognition'. | |
| gh_token | No | Optional GitHub token for higher rate limits. Falls back to the GITHUB_TOKEN env var. | |
| hf_token | No | Optional Hugging Face token for higher rate limits. Falls back to the HF_TOKEN env var. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and does well by disclosing that it returns deduplicated HarvestItem[], works without tokens, and is subject to public rate limits. It does not detail rate-limit specifics or clarify what 'sweep the 12-expert atlas' entails, but the core behavioral traits are transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences deliver the action, scope, output type, and authentication posture with no filler. The most important information is front-loaded, and every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the core call requirements: topic, optional tokens, output type, and deduplication. However, there is no output schema, and the description does not explain the structure of HarvestItem[], pagination, or the meaning of the '12-expert atlas,' leaving some gaps for an agent deciding whether this tool is sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all three parameters. The description adds useful context about plain-English topics and token-free operation, but it does not meaningfully extend parameter semantics beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies a specific verb ('Search') and resources ('Hugging Face datasets and GitHub repos'), with a plain-English topic as input. It also mentions sweeping the '12-expert atlas' for breadth, which gives a sense of scope, though it does not explicitly differentiate itself from sibling tools like harvest.expert.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this is the tool for broad, plain-English searches across multiple sources, especially when breadth is desired. However, it provides no explicit guidance on when not to use it or which sibling tool to choose instead, leaving the agent to infer the boundary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0harvest.composeharvest.expertharvest.latestharvest.packharvest.searchScored across 5 tools
The retrieval tools (search, expert, latest) are distinct in their modes: general search, expert-curated queries, and trending items. The composition tools (compose, pack) overlap slightly as both create mixes from a catalog, but pack adds a specific output format with coverage scores, reducing ambiguity.
All tool names are single lowercase verbs (search, expert, latest, compose, pack) following a uniform pattern. No mixing of styles or inconsistent verb forms.
With 5 tools covering search and composition workflows, the count is well-scoped and each tool serves a clear purpose. It is within the ideal 3-15 range and feels complete for the domain.
The tool set covers the core lifecycle: retrieving items via search, expert, and latest, then composing and packaging them. Minor gaps exist like no direct item detail fetch, but the returned HarvestItem[] likely includes necessary metadata, so workflows are not blocked.
Search GitHub, npm, PyPI, StackOverflow, ArXiv from one MCP — built for coding agents.
Search, sample and query open reproducible datasets published as immutable Parquet with schemas.
Verified open AI artifact search, trust evidence, downloads, and agent workflows from Hugging Bay.
Search and fetch AI agent skills, rules files and MCP servers indexed from GitHub.