"Web crawling techniques and tools" matching MCP servers:

An MCP-native pipeline for collecting structured intelligence on higher education institutions using the WHED schema, enabling scraping, extraction, validation, and saving of profiles.

Connect AI assistants to your Gramps Web family tree. Search and edit genealogy records, explore ancestors, descendants, relationships and timelines, and read document scans with vision-capable models. Includes guided research workflows, source citations, read-only mode, and privacy-aware media previews. Install with Docker or a Claude Desktop bundle.

Enables AI agents to search and fetch high-quality information from multiple sources, including general web APIs, Wikipedia, arXiv, Hacker News, Stack Exchange, and Crossref, with optional pro-mode deep research and clean markdown page extraction.

Enables querying MITRE ATT&CK and D3FEND data efficiently, providing techniques, detections, mitigations, actors, and countermeasures with token-aware summaries and paginated graph traversal.

An MCP server for global tech intelligence gathering, offering tools for searching Chinese and English web content, GitHub trending, arXiv papers, HN, Exa, SEC filings, Juejin, Zhihu, Bilibili, and A-share financial reports, all without API keys and with direct China access.

Enables managing Zotero libraries via the official Web API v3, including searching, creating, editing, and deleting items, collections, tags, notes, attachments, and performing batch operations, syncs, and file uploads/downloads.

Presents a plausible, tool-rich workspace surface — file reading, credential listing, command execution, database queries, and configuration retrieval — that returns only synthetic results while logging every connection, tool call, and argument received. It enables researchers to measure what MCP clients and agents actually do when handed dangerous-sounding tools, including whether they follow injected instructions or complete a retrieve-then-use credential chain.

Provides five rigorous reasoning protocols (debate, red team, audit_argument, threat_model, check_study) that run on the AI you're already using, requiring no extra API keys or costs.

Provides tools for AI-powered graph analysis, including relationship extraction, adjacency matrix creation, and network centrality calculations. It enables users to perform complex structural analysis and generate interactive D3.js visualizations from structured data.

Enables fetching any public web page through managed proxies, with optional JavaScript rendering, returning markdown, text, HTML, or structured JSON.

Provides actionable financial intelligence tools for AI agents including insider buying signals, earnings IV plays, market pulse, stock analysis, and options strategies via free public data sources.

An MCP server for managing pandoc/CSL-YAML bibliographies and verifying citations in documents.

High-value scientific tools for AI agents — literature search (PubMed, arXiv, Semantic Scholar), chemical compound lookup (PubChem, ChEMBL), patent prior art search (USPTO, EPO), GPU spot prices across 4 providers, and real-time earth science data (USGS, NASA, OpenAQ). Per-call billing via API key. Keys issued instantly at https://mcp-site.com/keys/request

An MCP server that enables users to read, search, and analyze PDF documents. It provides tools for extracting text, viewing metadata, searching content with context, and generating word statistics.

Enables querying real-time adoption metrics, trends, and comparisons for popular AI coding tools like OpenAI SDK, Anthropic SDK, Cursor, GitHub Copilot, and LangChain through natural language.

An MCP server that provides comprehensive financial insights and analysis by leveraging real-time market data, news, and advanced analytics for stocks, options, financial statements, and economic indicators.

Enables an agent to run ten ready-made research and monitoring workflows — competitor digests, cited reports, paper alerts, market maps, regulatory and due-diligence watches — pulling from Firecrawl, Tavily, GitHub and landing results in Notion, Slack, Google Docs, Sheets and Linear. Fields are filed with the user's own keys held in the operating system keychain, and every action that creates, sends or changes anything is shown and approved first.

Enables AI assistants to perform real-time web and academic searches using Perplexity's Sonar API.

Provides honest A/B test verdicts using interactive cards, with statistical calculations for conversion rates, sample sizes, and Bayesian probability-to-beat.

Enables AI-driven COMSOL batch jobs on the Harvard FASRC cluster, allowing users to run COMSOL models, manage jobs, and fetch results directly through natural language.