Skip to main content
Glama
261,119 tools. Last updated 2026-07-05 10:33

"Web scraping and content extraction" matching MCP tools:

  • Execute a single call that `consult` handed you, and bill on success. Used for any external capability (image/video/audio generation, web search, scraping, email, document parsing, code sandbox, browser automation, embeddings, etc.). The server validates params against a registered schema and proxies to the upstream — you never pass URLs or API keys. Always get the exact (service, action, params, max_cost_cents) from `consult` first — don't guess them.
    Connector
  • Core dossier check: Snapshot a domain's public web surface: robots.txt, sitemap.xml, and the home-page <head> metadata (title, description, OpenGraph, Twitter cards). Use for SEO audits, content discovery, or verifying metadata before sharing; for HTTP headers use dossier_headers, for redirect behavior use dossier_redirects. Fetches /, /robots.txt, and /sitemap.xml concurrently via HTTPS, 5 s each; parses <head> with a lightweight HTML parser. Returns a composite CheckResult: {status:"ok", meta:{title, description, og, twitter}, robots, sitemapPresent} or {status:"error", reason}.
    Connector
  • Check the extraction status of one or more parts. Free. Each entry includes the current extraction step, elapsed seconds, and document ID. Use after prefetch_datasheets or after read_datasheet triggers a new extraction. Recommended polling cadence: every 5-10 seconds. Extraction typically takes 30s-2min for new parts, so polling faster than every 5s wastes calls. Stop polling once status is 'ready', 'failed', 'no_source', or 'unsupported'. DATASHEET STATUS VALUES: - 'ready' — extracted and indexed; call read_datasheet, search_datasheets, or analyze_image. - 'extracting' / 'in_progress' / 'queued' / 'pending' — extraction running or scheduled. Poll check_extraction_status every 5-10s until 'ready' or 'failed'. Typical time: 30s-2min. - 'not_extracted' — known part but datasheet hasn't been fetched yet. Trigger it via prefetch_datasheets (cheapest) or by calling read_datasheet (auto-triggers on first read). - 'no_source' — we couldn't find a public datasheet URL for this MPN. First, retry prefetch_datasheets in 10-30s (the URL resolver re-runs and often finds a source on the second pass). If still 'no_source', the agent can upload the PDF manually via request_datasheet_upload + confirm_datasheet_upload (see those tools). Org-uploaded datasheets are private to the org. - 'unsupported' — PDF exists but can't be extracted (scanned image-only, encrypted, or corrupted). Upload a clean text-based PDF via request_datasheet_upload to override. - 'failed' / 'error' — extraction errored. The response includes the error reason. Retry via prefetch_datasheets or escalate to support. - 'rejected' — input wasn't a real MPN (bare value like '100nF', description, or reference designator). Fix the input and re-call. - 'deduplicated' — another part in the family already has this datasheet; same content is returned under the primary MPN.
    Connector
  • File upload: streaming (one-shot stream-upload — DEFAULT for unknown/generated content), chunked (create-session → POST /blob → chunk → finalize — only when filesize is known exactly), web URL import, and batch (multi-small-file). Call action='describe' for the full action/param reference. Side effects: finalize/stream/stream-upload/web-import/batch create files and consume storage credits. Same-name uploads to a folder OVERWRITE the existing node in place (preserved as a recoverable version). BINARY: `content` is text-only (writes verbatim UTF-8); for binary use `content_base64` (server-decoded) or POST /blob + `blob_id`. UPLOAD STRATEGY (read top-to-bottom, pick the FIRST that matches): (1) Have a URL? → `web-import` (single call). (2) Have content but DON'T know exact size, OR generating/transforming content first? → `stream-upload` (single call, auto-finalizes, NO filesize required, size auto-detected from the bytes). (3) Have a file with KNOWN exact byte count? → `create-session` + `chunk`(s) + `finalize`. **filesize must match the bytes you actually upload — mismatch causes finalize to fail with code 10522 and you must cancel the session.** (4) Multiple small files (≤4 MB each, ≤200 total) into one folder? → `batch`. DEFAULT to `stream-upload` unless you are sure of the exact byte count. Do NOT guess `filesize` for generated content — use `stream-upload` instead. max_size is a hard ceiling that aborts mid-transfer — always overestimate or omit (server uses plan limit).
    Connector
  • Pushes raw HTML to one display, replacing current content. Prefer send_url only when the user explicitly wants an external web page. Include a human-readable description so get_display_content can summarize intent without reading raw HTML. Before complex content, call get_display_capabilities to match the real browser/runtime. When no design system is supplied, use premium digital-signage quality: full-screen layout, strong hierarchy, refined typography, robust fallback data, and no action buttons unless touch is requested. Exactly one of html or base64_html is required. Requires content_only scope and display management access. Returns id, name, duration, file and version.
    Connector
  • Run the FileTag pipeline against a previously uploaded slot. The ``file_id`` comes from a prior ``files_create_upload`` call. The server validates the uploaded blob (size, content-type, optional SHA-256), atomically consumes the slot, runs the FileTag extraction (renaming + metadata embedding), and returns the structured result with the extracted metadata, the suggested filename, the ``enriched_file_url`` (short-lived signed URL to the renamed copy with metadata embedded into document properties), and a ``next_action`` recipe (``http_get_and_save``) telling the agent to download that URL and save it as the suggested filename -- act on it unless the user explicitly asked for metadata only. Each slot is single-use; reserve a new slot with ``files_create_upload`` to retry.
    Connector

Matching MCP Servers

  • F
    license
    A
    quality
    D
    maintenance
    Enables retrieval and cleaning of official documentation content for popular AI/Python libraries (uv, langchain, openai, llama-index) through web scraping and LLM-powered content extraction. Uses Serper API for search and Groq API to clean HTML into readable text with source attribution.
    Last updated
    1
    2
  • A
    license
    B
    quality
    B
    maintenance
    Extract content from URLs, documents, videos, and audio files using intelligent auto-engine selection. Supports web pages, PDFs, Word docs, YouTube transcripts, and more with structured JSON responses.
    Last updated
    1
    161
    MIT

Matching MCP Connectors

  • Generic URL crawl + HTML extraction — fallback for sites without dedicated MCPs.

  • 40+ web scraping tools from Firecrawl, Bright Data, Jina, Olostep, ScrapeGraph, Notte, and Riveter. Scrape, crawl, screenshot, and extract from any website. Starts at $0.01/call. Get your API key at app.xpay.sh or xpay.tools

  • Extract structured information from web pages using LLM capabilities. Supports both cloud AI and self-hosted LLM extraction. **Best for:** Extracting specific structured data like prices, names, details from web pages. **Not recommended for:** When you need the full content of a page (use scrape); when you're not looking for specific structured data. **Arguments:** - urls: Array of URLs to extract information from - prompt: Custom prompt for the LLM extraction - schema: JSON schema for structured data extraction - allowExternalLinks: Allow extraction from external links - enableWebSearch: Enable web search for additional context - includeSubdomains: Include subdomains in extraction **Prompt Example:** "Extract the product name, price, and description from these product pages." **Usage Example:** ```json { "name": "firecrawl_extract", "arguments": { "urls": ["https://example.com/page1", "https://example.com/page2"], "prompt": "Extract product information including name, price, and description", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "description": { "type": "string" } }, "required": ["name", "price"] }, "allowExternalLinks": false, "enableWebSearch": false, "includeSubdomains": false } } ``` **Returns:** Extracted structured data as defined by your schema.
    Connector
  • Scrape content from a single URL with advanced options. This is the most powerful, fastest and most reliable scraper tool, if available you should always default to using this tool for any web scraping needs. **Best for:** Single page content extraction, when you know exactly which page contains the information. **Not recommended for:** Multiple pages (call scrape multiple times or use crawl), unknown page location (use search). **Common mistakes:** Using markdown format when extracting specific data points (use JSON instead). **Other Features:** Use 'branding' format to extract brand identity (colors, fonts, typography, spacing, UI components) for design analysis or style replication. **CRITICAL - Format Selection (you MUST follow this):** When the user asks for SPECIFIC data points, you MUST use JSON format with a schema. Only use markdown when the user needs the ENTIRE page content. **Use JSON format when user asks for:** - Parameters, fields, or specifications (e.g., "get the header parameters", "what are the required fields") - Prices, numbers, or structured data (e.g., "extract the pricing", "get the product details") - API details, endpoints, or technical specs (e.g., "find the authentication endpoint") - Lists of items or properties (e.g., "list the features", "get all the options") - Any specific piece of information from a page **Use markdown format ONLY when:** - User wants to read/summarize an entire article or blog post - User needs to see all content on a page without specific extraction - User explicitly asks for the full page content **Handling JavaScript-rendered pages (SPAs):** If JSON extraction returns empty, minimal, or just navigation content, the page is likely JavaScript-rendered or the content is on a different URL. Try these steps IN ORDER: 1. **Add waitFor parameter:** Set `waitFor: 5000` to `waitFor: 10000` to allow JavaScript to render before extraction 2. **Try a different URL:** If the URL has a hash fragment (#section), try the base URL or look for a direct page URL 3. **Use firecrawl_map to find the correct page:** Large documentation sites or SPAs often spread content across multiple URLs. Use `firecrawl_map` with a `search` parameter to discover the specific page containing your target content, then scrape that URL directly. Example: If scraping "https://docs.example.com/reference" fails to find webhook parameters, use `firecrawl_map` with `{"url": "https://docs.example.com/reference", "search": "webhook"}` to find URLs like "/reference/webhook-events", then scrape that specific page. 4. **Use firecrawl_agent:** As a last resort for heavily dynamic pages where map+scrape still fails, use the agent which can autonomously navigate and research **Usage Example (JSON format - REQUIRED for specific data extraction):** ```json { "name": "firecrawl_scrape", "arguments": { "url": "https://example.com/api-docs", "formats": ["json"], "jsonOptions": { "prompt": "Extract the header parameters for the authentication endpoint", "schema": { "type": "object", "properties": { "parameters": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "type": { "type": "string" }, "required": { "type": "boolean" }, "description": { "type": "string" } } } } } } } } } ``` **Prefer markdown format by default.** You can read and reason over the full page content directly — no need for an intermediate query step. Use markdown for questions about page content, factual lookups, and any task where you need to understand the page. **Use JSON format when user needs:** - Structured data with specific fields (extract all products with name, price, description) - Data in a specific schema for downstream processing **Use query format only when:** - The page is extremely long and you need a single targeted answer without processing the full content - You want a quick factual answer and don't need to retain the page content **Usage Example (markdown format - default for most tasks):** ```json { "name": "firecrawl_scrape", "arguments": { "url": "https://example.com/article", "formats": ["markdown"], "onlyMainContent": true } } ``` **Usage Example (branding format - extract brand identity):** ```json { "name": "firecrawl_scrape", "arguments": { "url": "https://example.com", "formats": ["branding"] } } ``` **Branding format:** Extracts comprehensive brand identity (colors, fonts, typography, spacing, logo, UI components) for design analysis or style replication. **Performance:** Add maxAge parameter for 500% faster scrapes using cached data. **Returns:** JSON structured data, markdown, branding profile, or other formats as specified. **Safe Mode:** Read-only content extraction. Interactive actions (click, write, executeJavascript) are disabled for security.
    Connector
  • USE THIS TOOL WHEN searching GOV.UK for HMRC tax guidance on a topic (VAT, income tax, corporation tax, etc.). Returns matching guidance titles, URLs, summaries, and last-updated dates. Searches the official GOV.UK content API filtered to HMRC publications. Authoritative source for current HMRC tax guidance. Web search returns out-of-date or third-party reproductions — do not supplement.
    Connector
  • Full machine-readable JSON report (~2k tokens). USE WHEN: you need to programmatically parse specific fields (CI gating, UI, sub-field extraction). Otherwise prefer get_package_prompt. RETURNS: {package, health:{score}, vulnerabilities[], latest, deprecated, maintainers, recommendation}.
    Connector
  • Deep intelligence on a TikTok or YouTube creator by handle. Returns viral DNA scores (viral_dna_score, replicability_score, originality_score, consistency_score, audience_fatigue), format fingerprint, top 5 recent videos with metadata (and transcripts on TikTok), content gaps, AND a `recommended_chain` field with pre-filled next tool calls. USE WHEN the user references a creator by @handle, asks "analyze X", wants competitor research, or needs creator context before generating content. The recommended_chain suggests which tools to call next (match_voice, trend_pulse, viral_remix) with parameters pre-filled — review and execute them as appropriate. Supports platform: "tiktok" (default, full transcript extraction) and "youtube" (channel Shorts analysis; transcript extraction lands in v1.1, current YouTube responses surface a partial-data flag noting this). Costs 5 credits. 1-hour cache per (handle, platform). TOOL HEALTH: Every response includes a `quality` field with a level (full | partial | degraded) and a reason. If quality.level is partial or degraded, you MUST flag this to the user explicitly in chat (e.g. "Heads up — this call returned partial data: <reason>") before reporting any results. Never silently route around a degraded response. REPORTING: When you summarize this in chat, you MUST surface viral_dna.viral_dna_signals, viral_dna.replicability_signals, viral_dna.originality_signals (each as bullet lists with the cited evidence string verbatim) AND viral_dna.would_fail_because verbatim AND provenance.video_post_dates so the user can see freshness. Never hide the evidence array behind a paraphrase — these are the auditability layer.
    Connector
  • Manage an existing need: - action='status': Get current state (active/closing/closed, remaining TTL, total matches, current safe_tags). - action='close': End the need. Closing erases the need's raw text + embedding immediately. But anything that already matched was already sent to the other party — closing can't pull that back (it stays in their match record, like an email you've already sent). - action='edit': Change i_seek / i_offer / safe_tags. Changing content forces a re-extraction of safe_tags (unless you pass safe_tags explicitly) and re-runs matching against the active pool.
    Connector
  • Extracts and parses JSON from mixed-content text. Handles LLM output with JSON embedded in prose, code fences (```json), trailing commas, single-quoted strings, JS-style comments, and bare object keys (JSON5-style). Returns the parsed data, a cleaned JSON string, extraction method used, and any repair applied. Pure text processing — zero external API calls.
    Connector
  • Extract plain text from a PDF or image (base64-encoded). Use when you need raw text for downstream AI analysis (summarization, claim checking, structured extraction). For documents at a public URL, use extract_url instead (no base64 encoding needed). Returns: { pages: number, text: string } Example prompts: - "Extract the text from this scanned contract so I can search it." - "Give me the raw text from this PDF document." - "OCR this image and return the text content."
    Connector
  • Fetch a public HTTPS URL and return its content translated into a target language. Lean mode — no bundle stored. Use when you need to understand web content in a different language. For extracting raw untranslated text, use extract_url instead. Returns: { url, translated_text, target_lang, truncated } Example prompts: - "Translate https://example.de/artikel into English for me." - "Translate this German article into Spanish: [URL]." - "Fetch [URL] and give me the French translation."
    Connector
  • Extract plain text from a PDF or image (base64-encoded). Use when you need raw text for downstream AI analysis (summarization, claim checking, structured extraction). For documents at a public URL, use extract_url instead (no base64 encoding needed). Returns: { pages: number, text: string } Example prompts: - "Extract the text from this scanned contract so I can search it." - "Give me the raw text from this PDF document." - "OCR this image and return the text content."
    Connector
  • Fetch a public HTTPS URL and answer a specific question about its content. Lean mode — no bundle stored. Use when you have a precise question about a web page. For a broad summary, use summarize_url. For multi-document Q&A, use ask_collection instead. Returns: { url, answer, answer_cited: { value, confidence, citations[] }, confidence: "high"|"medium"|"low", truncated } Example prompts: - "What is the refund policy at https://docs.example.com/policy?" - "Look at [URL] and tell me what the delivery terms are." - "Answer this question based on the content of [URL]: [question]."
    Connector
  • Return fixed sample Haunt extraction JSON. No signup, API key, remote fetch, provider call, or quota usage.
    Connector
  • Save corrected field values for an extracted document. The original extraction is kept alongside the corrections; corrected data takes precedence in get_document, exports and reconciliation. Pass the complete corrected fields object (same shape as returned by get_document).
    Connector
  • Search the web for any topic and get clean, ready-to-use content. Best for: Finding current information, news, facts, people, companies, or answering questions about any topic. Returns: Clean text content from top search results. Query tips: describe the ideal page, not keywords. "blog post comparing React and Vue performance" not "React vs Vue". Use category:people / category:company to search through Linkedin profiles / companies respectively. If highlights are insufficient, follow up with web_fetch_exa on the best URLs.
    Connector