mcp-research

MCP server for web research, academic papers, Twitter/X, YouTube, and file ingestion. Eight tools for AI assistants — all via the MCP stdio protocol. Includes credential vault for institutional access, CAPTCHA detection, and token-efficient output.

Tools

All tools are read-only — they fetch and transform content, never modify anything.

Install

pip install mcp-research

Or run directly with uvx (zero-install):

uvx mcp-research

Optional extras:

pip install 'mcp-research[twitter]'    # yt-dlp for Twitter extraction
pip install 'mcp-research[youtube]'    # yt-dlp + faster-whisper for YouTube
pip install 'mcp-research[academic]'   # PyPDF2 for academic PDFs
pip install 'mcp-research[ingest]'     # PDF, DOCX, XLSX, PPTX, audio support
pip install 'mcp-research[all]'        # everything

Check your setup:

mcp-research doctor

Usage with Claude Code

Add to your Claude Code MCP config (~/.claude/settings.json or project .mcp.json):

{
  "mcpServers": {
    "research": {
      "command": "uvx",
      "args": ["mcp-research"],
      "env": {
        "BRAVE_API_KEY": "BSA...",
        "OLLAMA_URL": "http://localhost:11434"
      }
    }
  }
}

Usage with Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "research": {
      "command": "uvx",
      "args": ["mcp-research"],
      "env": {
        "BRAVE_API_KEY": "BSA..."
      }
    }
  }
}

Configuration

All configuration is via environment variables — no config files needed (except the optional vault).

Tool Details

web_search

web_search(query, max_results=5, summarize=False, auto_fetch_top=False)

Searches the web using a 3-tier cascade for maximum reliability:

1. Brave Search API — fast, high quality (requires BRAVE_API_KEY)

2. DuckDuckGo library — no API key needed, retries on rate limit

3. DuckDuckGo HTML scraper — last-resort fallback

Options:

• summarize: Use Ollama to summarize results (requires running Ollama)

• auto_fetch_top: Also fetch and return the full content of the top result

fetch_url

fetch_url(url, summarize=False, max_chars=15000)

Fetches a URL and converts it to clean markdown:

• SSRF protection: Blocks localhost, private IPs, non-HTTP schemes

• Smart retry: Exponential backoff on 429/5xx, per-hop redirect validation

• 24h cache: SHA-256 keyed, configurable TTL

• Content support: HTML → markdown, JSON → code block, binary → rejected

• Smart truncation: Breaks at heading/paragraph boundaries, not mid-text

• CAPTCHA detection: Flags Cloudflare, hCaptcha, reCAPTCHA, Akamai walls

• Token-efficient: Default 15K chars (~4K tokens), adjustable via max_chars

research

research(query, depth="standard", context="")

Compound research pipeline:

1. Query rewrite — Ollama optimizes your question into search keywords

2. Web search — finds relevant pages (with zero-result retry expansion)

3. Parallel fetch — fetches top N pages concurrently

4. Summarize — Ollama summarizes each page

5. Synthesize — Ollama produces a final cited answer

Depth levels:

All steps gracefully degrade without Ollama — you still get search results and page content.

youtube_essence

youtube_essence(url, mode="standard")

Extracts structured content from YouTube videos:

• Transcript: Auto-subtitles or Whisper transcription (local, private)

• Summary: AI summary via Ollama

• Key points: Bullet-point takeaways

• Chapters: Timestamped segments

• Quotes: Notable quotations (deep mode)

Modes: quick (TL;DR), standard (+ chapters), deep (+ quotes)

Requires yt-dlp. Optional: faster-whisper for audio-only videos, ffmpeg for media extraction.

deep_ingest

deep_ingest(path, include_types="", max_files=200, summarize=False)

Extracts text from files in a directory or single file:

• Text files: .txt, .md, .json, .csv, source code, etc.

• PDF: Via PyPDF2 (optional dependency)

• Office: .docx, .xlsx, .pptx (optional dependencies)

• Audio/Video: Whisper transcription (optional)

• Images: OCR via Ollama vision model (optional)

Type filter: text, pdf, audio, video, image, office

academic_lookup

academic_lookup(identifier, fetch_fulltext=True)

Resolves academic papers from multiple identifier types:

• DOI: 10.xxxx/... → Crossref metadata + publisher redirect

• ArXiv: 2301.12345 → abstract + PDF

• PubMed: PMID → E-utilities metadata → DOI chain

• URL: Publisher page detection

Full text access via credential vault:

• EZproxy rewriting (prefix and suffix modes)

• Bearer token, API key, basic auth, cookie jar

• Automatic publisher detection (IEEE, Springer, Elsevier, ACM, Wiley, Nature, JSTOR, etc.)

twitter_extract

twitter_extract(url, include_thread=False)

Extracts tweets and threads from X.com/Twitter using a strategy cascade:

1. yt-dlp (primary) — works with cookie jar for authenticated access

2. Twitter API v2 — if bearer token configured in vault

3. HTML fetch — cookie-based last resort

Returns: text, author, timestamp, metrics (likes, retweets, replies), media URLs.

vault_status

vault_status()

Shows loaded credential profiles, match patterns, and auth types — never exposes secrets. Also checks availability of optional dependencies.

Credential Vault

Create ~/.mcp-research/vault.yaml to configure authentication for protected sources:

version: 1
profiles:
  # University EZproxy for IEEE
  ieee-university:
    match: "*.ieee.org/**"
    ezproxy:
      base_url: "https://ezproxy.myuniversity.edu/login?url="
      mode: prefix

  # Springer via API key
  springer:
    match: "*.springer.com/**"
    auth:
      type: api_key
      header: "X-ApiKey"
      value: "${SPRINGER_API_KEY}"

  # X.com via browser cookies
  twitter:
    match: "*.x.com/**"
    auth:
      type: cookie_jar
      path: "${HOME}/.mcp-research/cookies/twitter.txt"

• ${VAR} resolved from environment variables — secrets never stored in plain text

• First matching profile wins (order matters)

• Auth types: bearer, basic, api_key, cookie_jar, headers

• EZproxy modes: prefix (prepend base URL) or suffix (domain rewriting)

• Hot-reload: vault file changes are picked up automatically

Token Efficiency

All tools produce compact output by default to avoid wasting AI context window tokens:

Safety & Robustness

• SSRF protection: Blocks localhost, private IPs, link-local, non-HTTP schemes on every hop

• CAPTCHA detection: Identifies Cloudflare, hCaptcha, reCAPTCHA, Akamai, DDoS-Guard walls

• Input validation: Size limits, URL validation, safe redirect following

• No eval/exec: No dynamic code execution

• Vault security: Secrets resolved from env vars, repr() redacts all auth values

• Cache isolation: Owner-only directory permissions (0o700)

• Graceful degradation: Missing optional deps don't crash — features degrade with clear messages

CLI

mcp-research serve                          # Run MCP stdio server (default)
mcp-research search "query"                 # Search the web
mcp-research fetch https://example.com      # Fetch URL to markdown
mcp-research youtube https://youtu.be/...   # Extract YouTube video
mcp-research ingest ./docs/                 # Extract text from files
mcp-research academic "10.1109/..."         # Resolve academic paper
mcp-research tweet https://x.com/.../123    # Extract tweet
mcp-research vault                          # Show vault profiles
mcp-research doctor                         # Check dependencies

Development

git clone https://github.com/MABAAM/Maibaamcrawler.git
cd Maibaamcrawler
pip install -e ".[all]"
pytest tests/ -v
python -m mcp_research

Changelog

v0.3.0

• Credential vault: YAML config at ~/.mcp-research/vault.yaml with env var interpolation, glob URL matching, EZproxy rewriting, hot-reload

• Session pooling: Per-domain sessions with vault auth injection, cookie jar support, idle eviction

• CAPTCHA detection: Identifies Cloudflare, hCaptcha, reCAPTCHA, Akamai, DDoS-Guard, generic bot walls

• Academic lookup: DOI/ArXiv/PubMed resolution, Crossref metadata, institutional full text access via vault

• Twitter/X extraction: yt-dlp, API v2, and cookie-based access with thread support

• Token efficiency: Default output caps (~4K tokens for fetch, ~500 per research source) to preserve AI context

• Doctor command: mcp-research doctor checks all dependencies and configuration

• Windows encoding fix: UTF-8 stdout/stderr wrapper prevents cp1252 crashes

v0.2.0

• YouTube essence: Transcript extraction, AI summary, key points, chapters, quotes

• Deep ingest: PDF, DOCX, XLSX, PPTX, audio, video, image text extraction

• Ollama integration: Query rewriting, summarization, synthesis, vision OCR

• Search logging: NDJSON event log for all operations

• Brave Search: Primary search tier with API key support

v0.1.0

• Initial release: 3 tools (web_search, fetch_url, research), SSRF protection, caching

License

MIT