topic-research
Uses Algolia's Hacker News search API to retrieve recent Hacker News stories about the topic for the news section.
Queries arXiv for recent papers matching the topic, returning them as references and a matching-paper count in trend signals.
Fetches DuckDuckGo search suggestions to generate keyword and related-topic ideas.
Searches GitHub repositories related to the topic, returning repositories as references and a matching-repository count in trends.
Optionally pulls Google Trends RSS and Google Suggest data for trending signals and keyword ideas, and can optionally use Apify's Google Search Scraper for web search results.
Optionally retrieves Google News RSS items related to the topic as an additional news source.
Pulls Lobsters' hottest stories and keeps only those that mention the topic as news.
Pulls trending links from mastodon.social and keeps only those that mention the topic as news.
Accepts user-supplied RSS/Atom feed URLs and includes matching feed items as news.
Queries the Stack Exchange API for relevant questions, returned as Q&A references.
Retrieves Stack Overflow questions via the Stack Exchange API as Q&A references.
Searches Wikipedia for related topics and an overview summary, and uses Wikimedia pageviews as trend signals including 7-day views and change versus the prior week.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Topic Research Pack
Give it a topic. Get back one JSON object with recent news, trend signals, keyword ideas and the clean markdown text of the top articles. Built for AI agents and RAG pipelines.
It runs three ways, all on the same core:
MCP server: any MCP-capable agent can call it as a tool (
research_topic,research_report,fetch_page_markdown,keyword_ideas).Apify Actor: run it on Apify, or call it from Apify's own MCP server once published.
CLI:
topic-research "your topic"for local use and debugging.
What you get
{
"topic": "vector databases",
"generatedAt": "2026-11-20T07:46:14.140Z",
"overview": {
"title": "Vector database",
"summary": "A vector database, vector store or vector search engine is a database that stores and retrieves embeddings of data in vector space. ...",
"url": "https://en.wikipedia.org/wiki/Vector_database",
"source": "wikipedia"
},
"news": [
{
"title": "I tried to beat \"the fastest vector database\" on its own benchmark",
"url": "https://kividb.io/benchmarks",
"source": "hackernews",
"publishedAt": "2026-11-03T22:32:20Z"
}
],
"trends": [
{ "label": "Wikipedia pageviews, last 7 days: Vector database", "value": 1311, "source": "wikipedia" },
{ "label": "Wikipedia pageviews change vs previous 7 days (%): Vector database", "value": 13.7, "source": "wikipedia" }
],
"keywords": [
{ "keyword": "vector databases list", "source": "duckduckgo-suggest" },
{ "keyword": "Milvus (vector database)", "source": "wikipedia" }
],
"references": [
{
"title": "Efficient Privacy-Preserving High-Dimensional Approximate Nearest Neighbor Search",
"url": "https://arxiv.org/abs/2611.25374v1",
"source": "arxiv",
"kind": "paper",
"publishedAt": "2026-11-19T02:13:01.000Z",
"summary": "In the AI era, approximate nearest neighbor (ANN) search on vector databases is critical a..."
},
{
"title": "milvus-io/milvus",
"url": "https://github.com/milvus-io/milvus",
"source": "github",
"kind": "repo",
"publishedAt": "2026-11-20T07:03:19Z",
"summary": "Milvus is a high-performance, cloud-native vector database built for scalable vector ANN s..."
}
],
"pages": [
{
"url": "https://kividb.io/benchmarks",
"title": "Benchmarks | KiviDB 1.2.0 against Redis, Valkey, Dragonfly and Garnet",
"markdown": "KiviDB 1.2.0 against Redis, Redis Cluster, Valkey, Dragonfly, Garnet and kvrocks: ..."
}
],
"sourcesUsed": ["hackernews", "wikipedia", "duckduckgo-suggest", "mastodon-trends", "lobsters", "arxiv", "stackexchange", "github"],
"errors": []
}(Trimmed from a real run. Real output has more items; page text is capped per page.)
overview is a short neutral description of the topic (the summary of the best-matching Wikipedia article; left out for ambiguous topics such as "Mercury"). generatedAt is the retrieval date that citations use. news is for news and trending links. references is for things to read that are not news: research papers, code repositories and Q&A threads (kind is paper, repo or question). trends also carries counts such as how many arXiv papers and GitHub repositories match the topic.
One source failing never fails the run. It is listed in errors and the rest of the pack is still returned.
Related MCP server: WebVector MCP Server
Sources
Source | Gives | Default |
Hacker News (Algolia API) | news, last 30 days | on |
Wikipedia search, summary + Wikimedia pageviews | related topics, the topic | on |
DuckDuckGo suggestions | keyword ideas (unofficial endpoint) | on |
Your RSS/Atom feeds | news (only items that mention the topic) | on when you pass feeds |
Mastodon trending links (mastodon.social) | news: trending links that mention the topic | on |
Lobsters hottest stories | news: stories that mention the topic | on |
arXiv |
| on |
Stack Exchange (Stack Overflow) |
| on |
GitHub repository search |
| on |
Google News RSS, Google Trends RSS, Google Suggest | news, trending-now signals, keyword ideas | off, see Google sources |
| web search results | off, Apify platform only, billed separately |
GDELT DOC API | news | off, experimental (see below) |
All "on" sources are public, need no API key, and make one request per run. Mastodon and Lobsters publish a global "what is trending" list that cannot be queried by topic, so only entries that mention all words of your topic are kept (plural-insensitive); for many topics that is nothing. Your own RSS feeds use the looser rule: any word of the topic. The unauthenticated GitHub and Stack Exchange APIs have small per-IP request limits, so on a busy shared network they can answer with a rate-limit error. That is reported in errors and does not fail the run. Skip any source with excludeSources (names as they appear in sourcesUsed), or set maxReferences to 0 to turn off arXiv, Stack Exchange and GitHub together.
GDELT is experimental. Every request to its API from the development machine was answered with HTTP 429, so its response format could not be checked against a real response. The parser follows the documentation of a maintained client library, and any mismatch shows up as an error in errors, not as wrong data. Enable it with enableGdelt (CLI --gdelt, MCP server --enable-gdelt).
How these sources were chosen, and what was rejected and why, is in project-management/sources-survey.md.
Quick start
Requires Node 22 or later.
npm install
npm run buildCLI
npx tsx src/cli.ts "ai agents"npx tsx src/cli.ts "ai agents" --pages 2 --feed https://example.com/feed.xmlOptions: --format json|markdown (default json), --feed <url> (repeatable), --pages <0-10>, --references <0-50>, --exclude <name,name>, --no-robots, --google, --gdelt.
--format markdown prints a readable brief instead of JSON:
npx tsx src/cli.ts "vector databases" --format markdown > brief.mdThe brief has an overview, signals, news, papers/code/questions, keyword ideas and short page excerpts. Every claim carries a numbered citation, and the Sources list gives each URL with its publication date and the retrieval date. It is assembled by fixed rules, not by a language model, and it says so at the end: check the originals before quoting them.
MCP server
Stdio (for local agents such as Claude Desktop or Claude Code). Use the absolute path to your checkout:
claude mcp add topic-research -- node /absolute/path/to/ResearcherAgent/dist/mcp-server.jsFor clients configured with JSON:
{
"mcpServers": {
"topic-research": {
"command": "node",
"args": ["/absolute/path/to/ResearcherAgent/dist/mcp-server.js"]
}
}
}Streamable HTTP (stateless, POST /mcp, binds to 127.0.0.1 by default):
node dist/mcp-server.js --http --port 3000Tools:
Tool | Use it to | Notes |
| the same research as a ready-to-read cited markdown brief | same args as |
| get the full pack for a topic | args: |
| read one URL as clean markdown | args: |
| get keyword and related-topic ideas fast | args: |
All four are read-only. Safety switches are server settings, not tool arguments, so an agent cannot change them: --no-robots (stop checking robots.txt), --enable-google (turn on Google sources) and --enable-gdelt (turn on the experimental GDELT source).
Apify Actor
Input:
Field | Default | Meaning |
| what to research | |
| 10 | news items to return (1-50) |
| 20 | keyword ideas to return (1-100) |
| 10 | papers, repositories and questions to return (0-50, 0 turns those sources off) |
| 3 | top articles to fetch as markdown (0-10, 0 turns it off) |
| 8000 | markdown is cut at this length |
| none | extra RSS/Atom feeds |
| none | source names to skip, for example |
| false | experimental GDELT news source |
| true | skip pages robots.txt disallows |
| false | see below |
| false | also call |
Output: one dataset item per run, shaped as above, plus the same research as a markdown brief in the run's key-value store under the key REPORT.md (not billed separately). Table views for the overview, news and keywords are defined in .actor/dataset_schema.json.
Run locally with the Apify CLI:
npx apify-cli run --purge --input-file input.jsonPricing (Apify Actor)
The Actor uses pay-per-event pricing. It charges two events, after the result is stored:
research-pack: once per successful run.page-markdown: once per page returned inpages.
A run in which every source failed is marked failed and is not charged. Your maximum-cost setting is respected: the run stops if one pack does not fit, and the number of pages is capped to what you can afford.
Prices are set in Apify Console and shown on the Actor's Pricing tab.
Turning on useGoogleSearchActor also runs apify/google-search-scraper, which is billed separately to your account.
Compliance and limits
Please read this before using it on other people's sites.
robots.txt is checked by default before every page fetch (RFC 9309 rules, including for redirect targets). A site with no robots.txt (404) is treated as open; if the file exists but cannot be fetched (server error, timeout), the page is skipped.
robots.txt is not permission. Some sites forbid automated use, scraping or use for AI in their terms of use even where robots.txt allows the page. You are responsible for following the terms of the sites you read.
Private addresses are blocked. Only
httpandhttpsare allowed. Every address a host resolves to, and every redirect, must be public. Requests are limited by time (10 s) and size (2 MB).Honest User-Agent. Requests identify as
TopicResearchPack.Pages that need JavaScript to render will come back mostly empty. Only the HTML the server sends is read.
Charset: the
Content-Typeheader is honoured;<meta charset>is not.
Google sources (off by default)
Google News RSS, Google Trends RSS and Google Suggest are unofficial endpoints. Google's terms restrict automated use, and the Google News feed states it is made available "for personal, non-commercial use" and that any other use is prohibited. Turning these on is your decision and your responsibility: check that your use is allowed before enabling them, especially for commercial use. They are off unless you set enableGoogleSources (Actor, CLI --google) or start the MCP server with --enable-google. This project never scrapes Google result pages directly.
Google News links are Google redirect URLs, so they are not fetched as pages.
HTTP transport
--http has no authentication. It binds to loopback and rejects other Host headers. Do not expose it beyond your machine without an authenticating proxy in front.
Development
npm run check # typecheck, lint, tests, build
npm test
npm run mcp # MCP server from source (stdio)
npm run cli -- "topic"Layout:
src/core/ research(), types (zod), sources/, content/ (robots + HTML to markdown), util/ (http, SSRF guard, RSS)
src/mcp/ MCP server and Streamable HTTP wrapper
src/actor/job.ts Actor logic (testable); src/actor.ts is the thin Apify binding
src/cli.ts CLI
.actor/ Actor definition, input and dataset schemas, Dockerfile
test/ node:test; the network is mocked with fixtures captured from real responses
project-management/ decisions, risks, progress, handoffsTests never touch the network. Fixtures in test/fixtures/ are small real API responses captured on 2026-11-15/16 or clearly labelled synthetic files; real news pages and feeds are deliberately not committed. See test/fixtures/README.md. Where a sample comes from documentation instead of a live response (GDELT, the apify/google-search-scraper output), the test says so.
Known gaps are tracked in project-management/risks/risks.md (DNS rebinding in the SSRF guard, unauthenticated HTTP mode, GDELT, Google terms).
License
MIT. See LICENSE. Changes are listed in CHANGELOG.md; contribution rules are in CONTRIBUTING.md and security reporting is in SECURITY.md.
The license covers this project's code. It does not grant any rights to third-party data or to the terms of the sites and APIs the tool reads; see Compliance and limits.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Web search, fetch, extract, and research for AI agents. Markdown output + AI-synthesized answers.
Official Exa MCP for web search, content fetching, and multi-step research.
Scrape, crawl and search the web for AI agents via MCP.
Live AI-native web search with citations. One tool for every MCP client. Flat per-request pricing.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to perform grounded web research with injection resistance, claim verification, and cost-aware routing through MCP tools like web_search, fetch_url, extract_claims, and check_grounding.MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to perform web research in one tool call by searching, fetching and cleaning full pages, and ranking cited passages. Provides web_research, web_fetch, web_search, and status tools for MCP clients.1MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to perform live web searches across 9 engines, scrape web pages into clean formats, and run agentic research with citations via MCP.60 PyPI6MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to perform web research over MCP: search with a real browser, fetch JS-rendered pages, download PDFs and convert them to Markdown, then search and page through large documents using bounded previews so the context window isn't flooded.BSD Zero Clause