Skip to main content
Glama
yubinkim444

ai-first-scraper-mcp

by yubinkim444
OverviewSchemaSourceRelated ServersScoreDiscussionsAdmin
Python
Remote

ai-first-scraper-mcp

Plug Claude Desktop, Cursor, or Cline straight into an ad-free web scraper + search engine. Three tools, one line of config.

PyPI Python MCP License: MIT

What it does

Adds three tools to any MCP-compatible agent:

Tool

What it does

fetch_page

Fetch one URL → return clean Markdown (HTML or PDF).

fetch_pages_batch

Fetch up to 25 URLs in parallel → return Markdown for each.

search_web

Run a web search and return the top-k result pages already converted to Markdown.

No more "the model called curl and then tried to parse 80kB of ad HTML." Your agent receives clean Markdown ready to reason about.

Backed by the ai-first-scraper and ai-first-search APIs.

Related MCP server: CC Web MCP

Install

Fastest — uvx (no install, runs from PyPI on demand)

// claude_desktop_config.json  /  cline_mcp_settings.json  /  ~/.cursor/mcp.json
{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "uvx",
      "args": ["ai-first-scraper-mcp"]
    }
  }
}

Restart your client (Claude Desktop / Cursor / Cline). The three tools above will appear automatically.

Alternative — pip install

pip install ai-first-scraper-mcp
{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "ai-first-scraper-mcp"
    }
  }
}

Where the config file lives

Client

Config path

Claude Desktop (macOS)

~/Library/Application Support/Claude/claude_desktop_config.json

Claude Desktop (Windows)

%APPDATA%\Claude\claude_desktop_config.json

Cursor

~/.cursor/mcp.json

Cline (VS Code)

~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json

Point at your own backend (optional)

By default this server calls the public ai-first-scraper.onrender.com and ai-first-search.onrender.com instances. If you want to self-host, set env vars in your MCP config:

{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "uvx",
      "args": ["ai-first-scraper-mcp"],
      "env": {
        "SCRAPER_URL": "https://your-scraper.example.com",
        "SEARCH_URL":  "https://your-search.example.com",
        "AFS_TIMEOUT": "60"
      }
    }
  }
}

Verify it works

Open your MCP client and ask the agent:

"Use the search_web tool to find the top 3 recent articles about MCP and summarize them in 5 bullets each."

You should see the agent call search_web, get back Markdown for each result, and produce the summary without ever touching raw HTML.

Companion projects

Develop locally

git clone https://github.com/yubinkim444/ai-first-scraper-mcp.git
cd ai-first-scraper-mcp

uv sync                    # or: pip install -e .
ai-first-scraper-mcp       # speaks MCP over stdio

To test against a local client, point its MCP config at the same command.

About the author

Built by yubinkim444, who also makes Kay's Records — an app for iOS and Android.

If this project saved you time, giving the app a try is the nicest way to say thanks.

License

MIT © yubinkim444

Available Tools

3 tools
fetch_pageA

Fetch a single web page or PDF and return its main content as clean, ad-free Markdown — ready to drop into an LLM prompt.

Args: url: A fully-qualified http(s) URL. max_tokens: Optional soft cap on the returned Markdown (whitespace tokens). When exceeded, the body is truncated and a [...truncated] marker is appended.

Returns: The cleaned Markdown body of the page.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
max_tokensNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries full burden. It discloses truncation behavior with max_tokens, return format (clean Markdown), and purpose. Missing error handling details, but core behavior is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: a single sentence plus clean Args and Returns sections. No fluff, every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema and sibling tools, the description provides enough context for typical use. Could mention error scenarios, but overall it's adequate for a single-page fetch tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description fully compensates by explaining url as fully-qualified http(s) URL and max_tokens as soft cap with truncation marker. Adds critical context beyond the schema types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it fetches a single web page or PDF and returns clean Markdown. It contrasts with sibling tools: 'single' vs batch, and URL-based vs search, providing clear differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use when you have a specific URL, but lacks explicit guidance on when to prefer this over fetch_pages_batch or search_web. No when-not or alternative scenarios are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_pages_batchA

Fetch many web pages in parallel and return each one's clean Markdown.

Use this whenever you need to read more than one URL at once — it is far faster than calling fetch_page in a loop because the upstream scraper handles the concurrency.

Args: urls: Up to 25 URLs. max_tokens: Optional per-URL soft cap on the returned Markdown.

Returns: A list of {url, ok, data?, error?} objects in the same order as the input URLs. data is {title, word_count, markdown, links, ...} on success; error contains the failure reason otherwise.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYes
max_tokensNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It declares concurrency (parallel), input limits (up to 25 URLs), per-URL soft cap (max_tokens), and output structure (ordered list with success/error). Missing: rate limits, auth, retries, timeouts. But for a batch fetch, these are sufficient to inform the agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is compact (~6 lines), with clear Args and Returns sections. No redundant or irrelevant information. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists (signal true), so return values are explained in description. Inputs fully described. Completeness is high for a straightforward batch fetch tool; minor omission: no explanation of potential error types or edge cases like invalid URLs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so description must compensate. It explains 'urls: Up to 25 URLs' and 'max_tokens: Optional per-URL soft cap on the returned Markdown.' This adds crucial constraints and semantics beyond the schema types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Fetch many web pages in parallel and return each one's clean Markdown.' It distinguishes from sibling tools: fetch_page (single page) and search_web (search). The verb 'fetch' and resource 'web pages' are specific, and the batch aspect is highlighted.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit guidance: 'Use this whenever you need to read more than one URL at once — it is far faster than calling fetch_page in a loop.' This directly tells when to use and why, contrasting with the sibling fetch_page.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_webA

Run a web search and return the top-k result pages already converted to clean Markdown. Use this whenever you need fresh information from the public web — it combines search and read in one call.

Args: q: The user's query (free text). k: How many results to fetch (1–10, default 5). max_tokens: Optional per-result soft cap on the returned Markdown.

Returns: A list of {url, title, snippet, ok, markdown, word_count, error?} result objects. Use title and snippet to decide which results are worth citing, then drop the markdown field into your prompt.

ParametersJSON Schema
NameRequiredDescriptionDefault
qYes
kNo
max_tokensNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description bears full burden. It describes the main behavior (search and convert to markdown) and return format, but does not disclose potential limitations like rate limiting, error handling, or result filtering details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear action sentence followed by detailed Args/Returns. It is slightly verbose but every sentence adds value. Front-loaded with purpose, then parameter details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (3 parameters, no nested objects), the description is sufficiently complete. It covers the return format, though the actual output schema is not in the prompt. Sibling tools provide context for selection.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description adds full semantic meaning for all parameters: q is 'free text', k is 'how many results (1-10, default 5)', max_tokens is 'optional per-result soft cap'. This fully compensates for missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states 'Run a web search and return the top-k result pages already converted to clean Markdown', providing a specific verb and resource. It distinguishes itself from sibling tools (fetch_page, fetch_pages_batch) by noting that it combines search and read in one call.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear guidance: 'Use this whenever you need fresh information from the public web'. It implies that siblings are for fetching known URLs, but does not explicitly state when not to use this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 3 tool updatesv1.0.0
    • First observedfetch_page
    • First observedfetch_pages_batch
    • First observedsearch_web
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.5/5.0

Scored across 3 tools

Disambiguation5/5

Each tool has a clearly distinct purpose: single page fetch, batch fetch, and web search. No overlap; agents can easily distinguish them.

Naming Consistency5/5

All tool names follow a consistent verb_noun snake_case pattern: fetch_page, fetch_pages_batch, search_web. The slight singularity/plural variation is minor and predictable.

Tool Count5/5

Three tools perfectly scope a scraping-focused server: single page, batch, and search. Neither too many nor too few for the domain.

Completeness5/5

The tool surface fully covers the core scraping workflow (fetch single, batch, search). No obvious gaps given the stated purpose of returning clean Markdown.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • Kryfto

    ExceptionRegret
    F
    license
    Not graded
    quality
    C
    maintenance
    Provides 42+ MCP tools for browser automation, web scraping, and search, enabling AI agents like Claude and Cursor to browse, extract data, and run research agents on the live web.
    8
    -

  • CC Web MCP

    JcDizzy
    A
    license
    Not graded
    quality
    D
    maintenance
    Provides web search and page fetching tools for third-party models (e.g., DeepSeek, Qwen, Kimi) in Claude Code, with configurable search backends, Markdown output, and safety boundaries.
    63
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/yubinkim444/ai-first-scraper-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server