PyScrappy
PyScrappy is a web scraping MCP server that provides AI agents with 22 tools to retrieve structured data from across the web — from general URLs to specialized platforms.
General Web Scraping
scrape_url— Scrape any URL for structured text, links, images, tables, and metadata; supports CSS selectors, pagination, and optional JavaScript rendering
Data & Research
scrape_wikipedia— Fetch Wikipedia articles in full, paragraph, or header modescrape_stock— Get Yahoo Finance stock quotes, historical price data, and company profilesscrape_news— Fetch articles from RSS/Atom feeds, auto-discover feeds from a news site, or extract full text from a single article URLsearch_images— Search for images and return URLs and metadata (default engine: Bing)search_youtube— Search YouTube videos and return titles, channels, links, and metadatasearch_linkedin_jobs— Search public LinkedIn job postings by keyword and locationsearch_github— Search GitHub repositories by query, sortable by stars, forks, or recencysearch_hackernews— Search Hacker News stories by relevance or datesearch_books— Search books via Open Library by title, author, or free textget_weather— Get current weather (temperature, humidity, wind, condition) for any location (no API key required)get_crypto— Get cryptocurrency prices, market cap, and 24h change via CoinGeckoconvert_currency— Get exchange rates and convert amounts between currenciesdefine_word— Look up English word definitions, part of speech, and usage exampleslookup_movie— Look up movie/TV info from IMDB via the OMDb API (requiresOMDB_API_KEY)
E-Commerce
search_amazon— Search Amazon products and return title, price, rating, and imagesearch_newegg— Search Newegg for electronics and computer hardwaresearch_ikea— Search IKEA furniture and home products with per-country pricing
Food Delivery
search_ubereats— List Uber Eats restaurants delivering in a cityget_ubereats_menu— Retrieve a full menu (items and prices) for a specific Uber Eats restaurantscrape_zomato— Search restaurants on Zomato by city (with optional cuisine/name filter)
Entertainment
search_soundcloud— Search SoundCloud tracks (uses browser backend for JS rendering)
Scrapes Amazon marketplace for product listings, including titles, prices, and details.
Scrapes GitHub repositories or profiles (details not fully shown in excerpt but listed as built-in scraper).
Scrapes IKEA product search results per country, including prices and details.
Fetches movie and TV information via OMDb API, such as title, year, rating, and genre.
Scrapes Newegg electronics and hardware product listings.
Scrapes RSS feeds (e.g., news articles) from any provided feed URL.
Searches SoundCloud for tracks and returns metadata like title and plays.
Scrapes Uber Eats restaurant listings and menus by city and locale.
Scrapes Wikipedia articles, summaries, and infoboxes by query.
Searches YouTube for videos and returns metadata such as title, views, and URL.
Scrapes Zomato restaurant listings by city.
PyScrappy is an AI-native web scraping toolkit that turns websites into structured, LLM-ready data. Use it as a Python library or expose it as an MCP server for AI agents.
📖 Documentation: pyscrappy.vercel.app
Key features
Generic scraper — give it any URL, get back structured text, links, images, tables, and metadata
LLM-ready output —
.to_markdown()turns any result into clean Markdown; also.to_json()and.to_dataframe()MCP server — expose the scrapers as tools for AI agents (Claude, Cursor, local LLMs, …)
JS rendering — optional Playwright backend for JavaScript-heavy sites
Custom selectors — pass CSS selectors to extract exactly what you need
Chainable
Selector— navigate HTML directly with CSS/XPath,find_all,find_by_text, andfind_similar(Scrapy/BeautifulSoup-style)Adaptive (self-healing) selectors — remember an element and relocate it by similarity when a site changes its markup, so scrapers don't silently break
Concurrent scraping —
scrape_many/scrape_allrun scrapes in parallelSitemap crawling — enumerate and scrape a whole site from its
sitemap.xml(index + gzip aware)Proxy & scraping-API support — route through a proxy or ScraperAPI/ScrapeOps for blocked sites
TLS-fingerprint impersonation —
impersonate="chrome"gets past anti-bot filters that block plain clients (optionalcurl_cffibackend)Command-line extract —
pyscrappy extract <url> out.mdscrapes a URL straight to a file, no codeRetry & rate-limiting — built-in exponential backoff and per-domain rate limiting
Type-safe — full type hints,
py.typedmarker20+ built-in scrapers — Wikipedia, IMDB, stocks, news, GitHub, Amazon/IKEA, YouTube, and more
Related MCP server: fetch
Installation
pip install pyscrappyOptional extras:
# Browser support (for JS-rendered pages)
pip install 'pyscrappy[browser]'
playwright install chromium
# DataFrame support
pip install 'pyscrappy[dataframe]'
# MCP server (use PyScrappy's scrapers as AI-agent tools)
pip install 'pyscrappy[mcp]'
# Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)
pip install 'pyscrappy[stealth]'
# Parquet / Excel export (ScrapeResult.to_parquet() / .to_excel())
pip install 'pyscrappy[parquet]'
pip install 'pyscrappy[excel]'
# Everything
pip install 'pyscrappy[all]'For AI agents
PyScrappy ships an MCP server that exposes its scrapers as tools, so an agent (Claude, Cursor, an OpenAI agent, a local LLM) can pull structured web data from any URL and hand it straight to the model:
AI agent ──MCP tool call──▶ PyScrappy ──fetch + extract──▶ Any website
▲ │
└────────────── clean Markdown / JSON ◀───────────────────────┘pip install 'pyscrappy[mcp]'
claude mcp add pyscrappy pyscrappy-mcpThen just ask: "use pyscrappy to summarize the latest headlines from bbc.com." See MCP server for the full setup and tool list.
Local models (Ollama), no MCP host needed
Ollama can't talk MCP on its own, so normally you'd run a host (Goose, Cline, …) in between. PyScrappy skips that with a built-in agent that talks to Ollama directly and lets a local model call the scrapers as tools:
pip install 'pyscrappy[mcp]' # needs Python 3.10+
pyscrappy chat --model qwen2.5 "what's the current AAPL quote?"It exposes the same 22 tools as the MCP server. The only requirement is a model
that supports tool calling (Llama 3.1, Qwen 2.5, Mistral, …); how well it
picks the right tool is up to the model. Point it at a remote Ollama with
--host, and pass -v to see each tool call.
MCP server (use PyScrappy from an AI agent)
PyScrappy ships an optional Model Context Protocol server, so an AI agent (e.g. Claude) can call PyScrappy's scrapers as tools and get structured web data back.
pip install 'pyscrappy[mcp]'The MCP extra installs the standalone fastmcp package and requires Python 3.10
or newer. On Python 3.9 the core scraping library still works, but the MCP server
is unavailable.
This installs the pyscrappy-mcp command. It uses stdio by default for local MCP
clients; Streamable HTTP and legacy SSE are available for remote deployments:
pyscrappy-mcp # stdio (default)
pyscrappy-mcp --http # Streamable HTTP
pyscrappy-mcp --sse # legacy SSEYou can also run the stdio server with python -m pyscrappy.mcp.
Register with Claude Code
claude mcp add pyscrappy pyscrappy-mcpRegister with Claude Desktop
Add to your claude_desktop_config.json and restart the app:
{
"mcpServers": {
"pyscrappy": {
"command": "pyscrappy-mcp"
}
}
}Tip: Claude Desktop does not inherit your shell
PATH. Ifpyscrappy-mcpis not found, use the absolute path to the command (e.g. the one printed bywhich pyscrappy-mcp).
Available tools
The server exposes 20+ tools. The most common ones are scrape_url (any
URL → text, links, images, tables, metadata), scrape_wikipedia,
scrape_stock, scrape_news, and search_github — plus many more
covering image/YouTube/LinkedIn/Hacker News/book search, weather, crypto,
currency, dictionary, Amazon/Newegg/IKEA/SoundCloud, IMDB, and Zomato/Uber Eats.
To see the full, live list, ask the agent to call the list_available_scrapers
tool, or from a shell:
python -c "from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))"The lookup_movie tool needs a free OMDb API
key. Pass it to the server through your MCP client config, e.g. for Claude Desktop:
{
"mcpServers": {
"pyscrappy": {
"command": "pyscrappy-mcp",
"env": { "OMDB_API_KEY": "your-key" }
}
}
}Once registered, just ask the agent naturally, e.g. "use pyscrappy to get the latest headlines from bbc.co.uk and the AAPL stock quote."
Built-in scrapers
PyScrappy ships 24 built-in scrapers, and every one that works without a proxy is also exposed as an MCP tool.
A few of them:
GenericScraper— scrape any URL with auto-extraction (text, links, images, tables, metadata)Data / research —
WikipediaScraper,StockScraper(Yahoo Finance),NewsScraper(RSS/Atom),GitHubScraper,HackerNewsScraper, plus weather, crypto, currency, dictionary, image, LinkedIn-jobs, and book searchE-commerce —
AmazonScraper,NeweggScraper,IKEAScraperSocial / media / food —
YouTubeScraper, SoundCloud, Zomato, Uber Eats (Instagram / Twitter / Spotify also ship, but are blocked and need a proxy)
…and many more. To see the full, live list:
python -c "from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))"IMDBScraper (lookup_movie) is the one exception that needs a key — a free
OMDb OMDB_API_KEY (see the
MCP config above for how to pass it).
Plugins
PyScrappy is extensible: you can add your own scrapers, and third parties can
ship them as standalone pyscrappy-<name> packages. A registered scraper works
everywhere a built-in does, including the MCP server and the pyscrappy chat
agent, with no change to PyScrappy core.
In your own code — register with the decorator:
from pyscrappy import BaseScraper, register_scraper, get_scraper
from pyscrappy.core.models import ScrapeResult, ScrapeMetadata
@register_scraper("reddit")
class RedditScraper(BaseScraper):
def scrape(self, subreddit: str, **kwargs) -> ScrapeResult:
data = self.fetch_and_parse(f"https://old.reddit.com/r/{subreddit}/.json")
# ... build a list of dicts ...
return ScrapeResult(data=[...], metadata=ScrapeMetadata(scraper="reddit"))
get_scraper("reddit")().scrape(subreddit="python")As a distributable package — advertise an entry point in your
pyproject.toml, and PyScrappy discovers it once your package is installed:
[project.entry-points."pyscrappy.scrapers"]
reddit = "pyscrappy_reddit:RedditScraper"After pip install pyscrappy-reddit, the scraper shows up in
list_scrapers(), and an AI agent can call it via the scrape_with MCP tool —
no core change required.
First-class MCP tools (optional). Add an mcp_tools mapping and your scraper
becomes a dedicated, typed MCP tool instead of only being reachable through the
generic scrape_with — its schema is derived from the method signature, so
agents get proper named arguments:
@register_scraper("reddit")
class RedditScraper(BaseScraper):
mcp_tools = {"search_reddit": "scrape"} # tool name -> method
def scrape(self, subreddit: str, sort: str = "hot") -> ScrapeResult:
...See the plugin template for a complete, copyable starting point, and the plugin guide for the full walkthrough.
Quick start
Scrape any URL → clean, LLM-ready Markdown
from pyscrappy import scrape
result = scrape("https://en.wikipedia.org/wiki/Web_scraping")
print(result.to_markdown()) # feed straight to an LLM
# ...or result.to_json() / result.to_dataframe()
# Write to a file — format inferred from the extension:
result.save("out.json") # .json .csv .md .ndjson .yaml .parquet .xlsx
# (.parquet needs pyscrappy[parquet]; .xlsx needs pyscrappy[excel])Prefer raw fields? Every result is a ScrapeResult with .data (a list of
dicts):
print(result.data[0]["metadata"]["title"])
print(result.data[0]["text"]["word_count"])Custom CSS selectors
from pyscrappy import GenericScraper
with GenericScraper() as gs:
result = gs.scrape(
url="https://news.ycombinator.com",
selectors={"title": ".titleline a", "score": ".score"},
)
for item in result.data:
print(item["title"], item.get("score", ""))Navigate HTML with Selector
When you want to traverse markup directly (Scrapy/BeautifulSoup-style) rather than
get back structured dicts, use Selector:
from pyscrappy import Selector
page = Selector(html) # or navigate any HTML string
page.css(".title::text").getall() # CSS with ::text / ::attr(name)
page.xpath("//a/@href").getall() # XPath (elements, text(), @attr)
page.find_all("h2", class_="title") # BeautifulSoup-style search
page.find_by_text("Add to cart", tag="button") # search by text content
first = page.css(".product")[0]
first.css(".price::text").get() # chainable
first.find_similar() # sibling elements shaped like this onecss() / xpath() return a SelectorList with .get() / .getall() / .text().
find_similar() locates elements with the same tag and overlapping classes, handy
for pulling every card/row once you've found one.
Adaptive (self-healing) selectors
A hard-coded CSS selector silently breaks the day a site changes its markup. Adaptive selectors survive that: save a fingerprint of the element the first time, and if the selector later matches nothing, relocate it by structural and textual similarity instead of returning empty.
from pyscrappy import Selector
# First run: match normally and remember this element under an id.
page = Selector(html_v1, url="https://shop.example.com")
price = page.css(".price", auto_save=True, adaptive_id="price").get()
# Later, after a redesign renamed ".price" — heal instead of breaking.
# `expect` is an optional contract: the relocated element must satisfy it,
# so a good structural score can't smuggle in the wrong field.
page = Selector(html_v2, url="https://shop.example.com")
result = page.css(
".price",
adaptive=True,
adaptive_id="price",
expect=lambda s: s.text().startswith("$"),
)
print(result.get(), "→ confidence:", result.adaptive_confidence)How the relocation decides — and where it's stronger than a naive similarity match:
Weighted signals, not a flat average. A stable
id/data-*hook counts far more than a sibling-tag list, so weak signals can't outvote strong ones.Anchor-relative. It remembers the nearest stable ancestor (an id'd /
data-*container) and depth, so it survives layout reshuffles that move absolute positions.Volatility-aware text. Prices, dates, and counts are down-weighted, so healing stays reliable on exactly the fields that change most between scrapes.
Confidence-scored.
SelectorList.adaptive_confidence(0-100) tells you how sure the relocation was;threshold=sets the minimum to accept.Contract-enforced (opt-in). Pass
expect=<callable>to require the healed element to satisfy an invariant (e.g. "text looks like a price"). A heal that clears the threshold but fails the contract is rejected, so structural similarity alone never redefines what a field means.
A heal is a change to what a selector resolves to, so every accepted heal is
recorded. The store keeps an append-only audit log (adaptive.heal.ndjson
beside the fingerprint store) with the confidence, the runner-up gap, and the
before/after fingerprint, readable via store.heal_log() — so drift stays
observable instead of being silently absorbed. For an at-a-glance summary,
store.heal_report() aggregates the log into one row per selector (heal count,
latest/lowest/average confidence, when it last healed), sorted most-healed first
— so the selectors that have drifted the most, and the shakiest relocations
(lowest confidence), surface at the top for a human to review.
Fingerprints persist in a small JSON store (~/.pyscrappy/adaptive.json by
default, or $PYSCRAPPY_HOME), namespaced by site so the same adaptive_id on
two sites never collides. Adaptive is entirely opt-in: without adaptive=True, a
broken selector still just returns empty, exactly as before.
Site-specific scrapers
Every built-in scraper follows the same pattern — instantiate, scrape(...),
read result.data (or .to_dataframe() / .to_markdown()):
from pyscrappy import WikipediaScraper
with WikipediaScraper() as ws:
result = ws.scrape(query="Python (programming language)", mode="summary")
print(result.data[0]["text"])Each scraper has its own arguments (Wikipedia, stocks, IMDB, news, YouTube, Amazon/Newegg/IKEA, Uber Eats, and more — see the full list). For per-scraper arguments and examples, see the documentation.
From the command line
Scrape a URL straight to a file without writing any code — the output format is inferred from the file extension:
pyscrappy extract https://example.com out.md # clean Markdown
pyscrappy extract https://example.com out.json # structured JSON
pyscrappy extract https://example.com out.txt # extracted page text
pyscrappy extract https://example.com out.html # raw fetched HTML
# Narrow to elements matching a CSS selector, or render JS first:
pyscrappy extract https://example.com items.txt --css-selector ".product"
pyscrappy extract https://example.com page.md --render-jsConfiguration
from pyscrappy import ScraperConfig, GenericScraper
config = ScraperConfig(
timeout=20.0, # request timeout in seconds
max_retries=3, # retry failed requests
retry_jitter=True, # spread exponential retries to avoid lockstep traffic
rate_limit=2.0, # seconds between requests per domain
proxy="http://...", # proxy URL, or a list to rotate through
scraper_api=None, # route via a scraping-API service (see below)
headless=True, # browser runs headless
render_js="auto", # auto-detect if JS rendering is needed
cache_ttl=0, # response cache TTL in seconds (0 = disabled)
cache_dir=None, # also persist the cache to disk (survives restarts)
cache_dir_max_size=512, # max live entries kept on disk before oldest are pruned
impersonate=None, # e.g. "chrome" to spoof a browser's TLS fingerprint (see below)
)
with GenericScraper(config) as gs:
result = gs.scrape(url="https://example.com")Proxies and blocked sites
Some sites (e.g. eBay, Instagram, Twitter/X, Spotify) block direct automated requests. PyScrappy supports two ways to get through them.
A proxy (or a rotating list) — applies to both the HTTP and browser backends:
from pyscrappy import ScraperConfig, AmazonScraper
# Single proxy
config = ScraperConfig(proxy="http://user:pass@host:port")
# Rotating list (one picked per request)
config = ScraperConfig(proxy=["http://p1:8080", "http://p2:8080"])A scraping-API service (ScraperAPI, ScrapeOps, ScrapingBee) — routes requests through the service, which handles proxies and anti-bot challenges for you:
config = ScraperConfig(scraper_api={
"provider": "scraperapi", # or "scrapeops", "scrapingbee"
"api_key": "YOUR_KEY",
"render_js": True, # optional
})
# Now any scraper works through the service, unchanged:
with AmazonScraper(config) as scraper:
result = scraper.scrape(query="laptop")This is the reliable way to use the scrapers marked "needs proxy" above.
TLS-fingerprint impersonation — many anti-bot systems block a plain HTTP
client by its TLS/JA3 fingerprint before serving any content. Set impersonate
to mimic a real browser's fingerprint and get past that class of block without a
headless browser:
from pyscrappy import ScraperConfig, GenericScraper
# needs the optional extra: pip install 'pyscrappy[stealth]'
config = ScraperConfig(impersonate="chrome") # or "chrome124", "safari", "firefox"
with GenericScraper(config) as gs:
result = gs.scrape("https://example.com")Impersonation works on both the sync and async paths (async uses
curl_cffi's AsyncSession), so you can combine stealth with high-throughput
async scraping. All the usual retry, rate-limiting, caching, and robots handling
still apply.
import asyncio
from pyscrappy import scrape_async, ScraperConfig
async def main():
cfg = ScraperConfig(impersonate="chrome")
return await scrape_async("https://example.com", config=cfg)
asyncio.run(main())Concurrent scraping
Scraping is I/O-bound, so running several scrapes at once parallelizes the
network waits. scrape_many runs one scraper over many inputs; scrape_all
runs a mix of scrapers together. Both preserve input order.
from pyscrappy import scrape_many, scrape_all, AmazonScraper, WikipediaScraper, NewsScraper
# One scraper, many queries, concurrently:
results = scrape_many(AmazonScraper, [{"query": "laptop"}, {"query": "phone"}])
# Different scrapers at once:
results = scrape_all([
lambda: WikipediaScraper().scrape(query="Python"),
lambda: NewsScraper().scrape(feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"),
])Sitemap crawling
Pagination follows next-page links; a sitemap enumerates a whole site's URLs
directly. GenericScraper can read /sitemap.xml (discovered from robots.txt
Sitemap: directives, or the conventional path), follow a <sitemapindex> into
its child sitemaps, and scrape every listed page.
from pyscrappy import GenericScraper
with GenericScraper() as gs:
# Just enumerate the URLs:
urls = gs.sitemap_urls("https://example.com") # -> list[str]
# Or fetch + extract each, concurrently, into one result:
result = gs.scrape_sitemap("https://example.com", max_urls=100)
print(len(result.data), "pages scraped")Handles <urlset> leaves and <sitemapindex> files (recursing one level),
gzip-compressed sitemaps (.xml.gz), and de-duplicates URLs. Fetches go through
the usual rate-limiting, caching, proxy, and stealth machinery, and the fan-out
reuses scrape_all. max_urls caps the crawl (a sitemap can list tens of
thousands of URLs, so it's required for scrape_sitemap).
Response caching
Set cache_ttl to a positive number of seconds to cache successful GET
responses. Repeated requests for the same URL (and query params) within the TTL
are served from cache, skipping both the network and the rate limiter. Caching
is disabled by default (cache_ttl=0).
from pyscrappy import WikipediaScraper
from pyscrappy import ScraperConfig
config = ScraperConfig(cache_ttl=300) # cache for 5 minutes
with WikipediaScraper(config) as ws:
ws.scrape(query="Python") # fetched over the network
ws.scrape(query="Python") # served from cacheThe cache is in memory and shared across scraper instances in the same process
(so it also speeds up repeated calls through the MCP server), and is cleared
when the process exits. Call HttpClient.clear_cache() to empty it manually.
It is LRU-bounded: at most cache_max_size live entries (default 512),
with the least-recently-used entry evicted once the cap is reached. So a
long-running process (e.g. the MCP server) that fetches many distinct URLs stays
bounded rather than growing until restart. Raise or lower the cap as needed:
config = ScraperConfig(cache_ttl=300, cache_max_size=2000)Persistent (on-disk) cache. Set cache_dir to also persist responses to
disk, so cache hits survive across process restarts and separate runs — useful
for re-running a scrape or a CLI job without re-fetching. The in-memory cache
still fronts it for speed; a disk hit is promoted back into memory.
config = ScraperConfig(cache_ttl=3600, cache_dir="~/.cache/pyscrappy")The on-disk cache is bounded too: each write prunes expired entries and trims the
oldest past cache_dir_max_size (default 512), so a cache_dir doesn't grow
one file per distinct URL forever.
clear_cache() empties the in-memory cache; the on-disk cache persists by
design — delete its cache_dir to clear it.
Observability hooks
For long crawls, pass lightweight callbacks to watch requests live (progress bars, metrics) without turning on logging:
config = ScraperConfig(
on_request=lambda url: print("GET", url), # before a network fetch
on_retry=lambda url, attempt, delay, err: print("retry", attempt, url),
on_cache_hit=lambda url: print("cached", url), # served from cache
)on_request(url)fires once before a URL is fetched (not on a cache hit).on_retry(url, attempt, delay, error)fires before each backoff sleep.on_cache_hit(url)fires when a request is served from cache.
All three are best-effort: a callback that raises is logged at debug and never breaks the scrape. They fire on both the sync and async paths.
Dependencies
Required: httpx, beautifulsoup4, lxml
Optional: playwright (JS rendering), pandas (DataFrames), fastmcp
(MCP server, Python 3.10+)
License
Contributing
All contributions welcome. See Issues.
This package is for educational and research purposes.
Available Tools
24 toolsconvert_currencyConvert CurrencyA
Fetch live exchange rates and convert an amount from one currency to others.
Returns a dict with the base currency, the amount converted, and a mapping of each target currency code to its converted value and unit exchange rate (e.g. {"base": "USD", "amount": 100, "results": {"EUR": {"rate": 0.92, "value": 92.0}}}).
Inputs: to, base, amount. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
define_wordDefine WordA
Look up an English word and return its dictionary entry: definitions, part(s) of speech, and example sentences.
Fetches from an online dictionary data source, so a network connection is required. Read-only with no side effects. If the word is not found (misspelled or not in the dictionary), returns an empty result or a not-found response rather than raising.
Returns a structured entry for the word, typically containing the word itself, one or more part-of-speech groupings, and for each a list of definitions with optional example sentences.
Inputs: word (required). Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
get_cryptoGet CryptoA
Fetch live cryptocurrency market data and return a list of coin records, each with fields: id, symbol, name, current price (in vs_currency), market cap, and 24h price change (percent).
Fetches from a live crypto market data API over the network, so results reflect current prices and require internet access; no local state is read or written. When query is omitted, returns the top coins ranked by market cap. If no coins match the query, returns an empty list rather than raising.
Inputs: query, max_results, vs_currency. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
get_weatherGet WeatherA
Fetch the current weather conditions for a named place and return a dict with keys: temperature (number, degrees Celsius), humidity (number, percent), wind_speed (number, wind speed), condition (str, e.g. "Clear", "Rain"), and location (str, the resolved place name).
Makes a live network call to an external weather provider on each invocation, so results reflect real-time conditions and require internet access. If the location cannot be resolved or the provider returns no match, the tool returns an empty result (or an error field) rather than raising.
Inputs: location (required). Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
list_available_scrapersList Available ScrapersA
List every scraper registered with this server and return their names for use with scrape_with.
Reads the server's in-process scraper registry, which includes built-in scrapers plus any installed third-party pyscrappy-* plugin packages that self-register on import. No network or browser access is performed and no state is changed. If no scrapers are registered, returns an empty list.
Returns: list[str]: Scraper name identifiers (for example ["amazon", "flipkart", "youtube"]), each usable as the scraper argument to scrape_with. Empty list when none are registered.
Usage Guidelines: Call this first to discover valid scraper names, then pass a returned name to scrape_with; use it to confirm a plugin registered correctly after installing a pyscrappy-* package.
Inputs: no named parameters. Returns: object
lookup_movieLookup MovieA
Look up movie and TV data from IMDB via the OMDb API and return a JSON-serializable dict; a title search returns {"results": [...]} with each item holding title, year, imdb_id, and type, while an IMDB-id lookup returns a single record with full details (plot, ratings, cast, runtime, genre).
Reads over the network from the OMDb HTTP API; no browser is needed and nothing is written or cached. Requires a free OMDb API key in the OMDB_API_KEY environment variable (get one at https://www.omdbapi.com/apikey.aspx); if it is missing the tool returns {"error": ...} explaining how to set it. When the query matches nothing, it returns an empty results list rather than raising.
Inputs: query (required), max_pages. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
scrape_newsScrape NewsA
Fetch news articles from an RSS/Atom feed, a news site (feed auto-discovered), or a single article, and return a list of article dicts (typically: title, url, published date, author, summary, and full text where available).
Provide exactly one of feed_url, site_url, or article_url. Fetches live content over the network at call time; results are not cached. article_url returns one article; feed_url and site_url return up to max_articles. Returns an empty list if the feed/site yields no articles or if a feed cannot be discovered or parsed.
Inputs: feed_url, site_url, article_url, max_articles. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
scrape_stockScrape StockA
Fetch stock market data from Yahoo Finance and return it as a dict.
The returned shape depends on mode:
"quote": {"symbol", "currency", "exchange", "price", "previous_close", "volume", "day_high", "day_low", "fifty_two_week_high", "fifty_two_week_low"}.
"history": {"symbol", "period", "rows": [{"date", "open", "high", "low", "close", "volume"}, ...]}.
"profile": {"symbol", "name", "currency", "exchange", "market", "timezone", "instrument_type"}.
Inputs: symbol (required), mode, period, interval. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
scrape_urlScrape UrlA
Scrape any HTTP(S) URL and return a ScrapeToolResult whose data holds one object per page containing extracted text (with word_count), links, images, tables, and page metadata.
Fetches the page over the network and parses the HTML; no data is stored or mutated. By default it makes a plain static HTTP request, so pages built client-side with JavaScript come back nearly empty. When that is detected, the returned errors list gets a hint to retry with render_js=true; set render_js=true to render with a headless browser instead (requires the pyscrappy[browser] extra). On empty or failed results, data is [], count is 0, and errors describes the problem rather than raising.
Inputs: url (required), max_pages, render_js, selectors. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
scrape_wikipediaScrape WikipediaA
Fetch a Wikipedia article by title or search term and return its text content.
Makes a live network request to Wikipedia, resolving the query to the best-matching article and extracting its body. The shape of the returned text depends on mode: "full" returns the entire article as one string; "paragraphs" returns the article split into a list of paragraph strings; "headers" returns a list of the article's section heading strings (its table of contents). If no article matches the query, an empty result is returned (empty string for "full", empty list for "paragraphs" or "headers").
Inputs: query (required), mode. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
scrape_withScrape WithA
Run any registered scraper (built-in or plugin) by name and return that scraper's raw scrape() output.
This is the generic dispatch entry point for scrapers that lack a dedicated tool, notably third-party plugins. It looks up the scraper in the registry, calls its scrape() method with the given args, and returns whatever that scraper returns (typically a dict or list of records; exact shape is scraper-specific). Side effects and requirements (network requests, browser/headless rendering, auth) depend entirely on the target scraper. If name is not a registered scraper, it raises an error rather than returning empty; if the scraper runs but finds nothing, it returns that scraper's empty result (e.g. an empty list).
Inputs: name (required), args. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
scrape_zomatoScrape ZomatoA
Search Zomato for restaurants in a city and return a list of restaurant records.
Scrapes Zomato's public restaurant listings over the network for the given city, optionally filtered by a cuisine or name term. Each result is a dict with fields such as name, cuisine, rating, price_for_two, address, and url; the exact keys depend on what Zomato exposes for each listing. Returns a list of these dicts ordered as Zomato ranks them, capped at max_results. Returns an empty list if the city is unknown or no restaurants match the query. Requires outbound network access; results reflect live Zomato data at call time and may vary between calls.
Inputs: city (required), query, max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_amazonSearch AmazonA
Scrape Amazon search results for a query and return a list of matching products, each with its title, price, rating, and image URL.
This performs a live network scrape of Amazon's public search results pages (no login, no API key). It has no side effects beyond outgoing HTTP requests. Results reflect Amazon's current listings and may vary by region, availability, and anti-bot throttling. Returns a list of dicts, one per product, each shaped as {"title": str, "price": str, "rating": str, "image": str}; fields that Amazon omits for a listing come back as empty strings or None. Returns an empty list when the query yields no products or when scraping is blocked.
Inputs: query (required), max_pages. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_booksSearch BooksA
Search books by title, author, or free text via the Open Library search API and return a list of matching book records.
Queries Open Library over the network (no browser or authentication required). Returns a list of dicts, each typically containing: title (str), author_names (list of str), first_publish_year (int or None), edition_count (int), and the Open Library work key (str, e.g. "/works/OL45804W"). Fields missing upstream are omitted or None. Returns an empty list when the query matches nothing. Read-only: no local files or state are modified.
Inputs: query (required), max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_githubSearch GithubA
Search GitHub for public repositories and return a list of repository records.
Queries the GitHub search API over the network and returns a list of dicts, each with: name (str), owner (str), stars (int), description (str), and language (str). Results are ordered per the sort argument. Returns an empty list when no repository matches the query. Requires network access; may be subject to GitHub API rate limits.
Inputs: query (required), sort, max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_hackernewsSearch HackernewsA
Search Hacker News stories and return a list of matching story dicts, each with title, url, points, author (username), and num_comments (comment count).
Queries the public Hacker News search index (Algolia HN API) over the network; makes no local changes. Returns an empty list when nothing matches or the query is empty.
Inputs: query (required), by, tags, max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_ikeaSearch IkeaA
Search IKEA's online catalog for furniture and home products, returning a list of product dicts with fields name, type, price, and rating.
Scrapes the IKEA store website for the given country at call time, so results require network access and reflect that store's live listings. Prices, availability, and currency are per-country and per-language. Returns an empty list if the query matches no products.
Inputs: query (required), lang, country, max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_imagesSearch ImagesA
Search the web for images and return a list of result objects with image URLs and metadata.
Each result is a dict with keys: "url" (direct link to the image), "thumbnail" (small preview), "title" (caption or alt text), "source_page" (page the image was found on), "width", and "height" (pixels). Every engine returns this same key set; fields a given engine can't provide are empty ("" for text, null for width/height — e.g. the Google path fills only "url", "title", and "source_page"). Results are returned in the engine's relevance order.
Inputs: query (required), engine, max_images. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_linkedin_jobsSearch Linkedin JobsA
Search LinkedIn public job postings and return a list of matched jobs.
Scrapes LinkedIn's public job search results over the network (no login required) and returns a list of dicts, one per posting, typically with keys: title, company, location, url, and posted_date. Returns an empty list if no postings match or the query yields no results. Live web scraping, so results reflect LinkedIn at call time and may vary between runs.
Inputs: query (required), location, max_pages. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_neweggSearch NeweggA
Search Newegg for electronics and computer hardware, returning a list of product dicts each with title, price, product_url, image_url, rating, and item_number.
Live-scrapes Newegg search result pages over the network; requires outbound internet access and returns an empty list if no products match or the page structure cannot be parsed. Read-only, with no side effects beyond the outbound HTTP requests.
Inputs: query (required), max_pages. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_soundcloudSearch SoundcloudA
Search SoundCloud for tracks and return a list of track dicts, each with keys: title (str), artist (str), plays (int), likes (int), and url (str, the track page URL).
Renders SoundCloud's JavaScript search results with a browser backend (Playwright/Selenium), so it requires the pyscrappy[browser] extra to be installed and launches a headless browser per call. This makes it slower and heavier than the HTTP-based search tools. Results reflect SoundCloud's live public search at call time; no login or API key is used. Returns an empty list if the query matches no tracks.
Inputs: query (required), max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_ubereatsSearch UbereatsA
Search Uber Eats for restaurants delivering in a given city, returning a ScrapeToolResult envelope whose data is a list of restaurant objects (typically name, eta, delivery fee, and store url).
Fetches live listings from Uber Eats over the network at call time; no API key is required. The data list is capped at max_results and each item's store url is the input for get_ubereats_menu. Alongside data, the envelope carries count, scraper, source_urls, and errors (non-fatal issues, each with a url and message). If the city is unrecognized or no restaurants are found, data is an empty list and count is 0.
Inputs: city (required), max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
search_youtubeSearch YoutubeA
Search YouTube for videos matching a query and return a list of matching videos with their metadata.
Performs a live YouTube search over the network, so results reflect current YouTube data and may vary between calls; it is read-only and has no side effects. Returns a list of video objects, each typically containing: title (str), channel (str), url/link (str) to the video, video_id (str), duration (str), view_count (int), and published/upload date (str). Returns an empty list when the query matches no videos.
Inputs: query (required), max_results. Returns: Structured result returned by every PyScrappy MCP tool. ``data`` holds the scra…
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 24 tools
Most tools are clearly separated by target source or data type, so an agent can usually pick the right one from the name alone. The main ambiguity is that scrape_with is a generic dispatch that can overlap with the specialized scrapers, and a few search/scrape pairs (e.g. search_ubereats vs scrape_zomato) serve similar purposes but descriptions disambiguate them.
Names follow readable snake_case verb_noun structure, but the verbs are mixed across three patterns: search_*, scrape_*, and get_* with occasional one-offs like lookup_movie, define_word, and scrape_with. This is consistent within clusters but not predictable across the whole toolset.
At 24 tools, the set is in the heavy range and feels like a broad aggregation of scrapers rather than a tightly scoped server. The count is defensible for a multi-site scraping toolkit, but it is large enough that navigation and selection become a real burden.
The server covers the core scraper lifecycle well: generic URL scraping, named-source scrapers, plugin discovery, and generic dispatch via scrape_with. Minor gaps exist, such as no runtime scraper registration and no detailed arg discovery for registered plugin scrapers, but the main workflows are functional.
Maintenance
Related MCP Connectors
Structured web research tool for AI agents: search, fetch and shape web data into the JSON schema…
Web data for agents: YouTube transcripts, screenshots, Google News, WHOIS, jobs, tech stack, more.
Public web data for AI agents: scrape any page, plus Google, TikTok, Instagram and Amazon as JSON.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceWeb scraping and search MCP server that wraps Firecrawl API for URL discovery and web search with optional content retrieval.165 npm1MIT
- AlicenseNot gradedqualityDmaintenanceFetches web pages and converts them to markdown for LLM consumption, supporting chunked reading and raw content extraction.MIT
- AlicenseAqualityBmaintenanceWeb tools for AI agents. Search the web for full page content, fetch URLs as clean markdown including PDFs, extract structured data from a page with a prompt, and run multi-source deep research that returns a cited report.42MIT
- FlicenseNot gradedqualityCmaintenanceEnables AI agents to search the web, read and extract content from webpages, fetch JSON from REST APIs, and collect links while bypassing anti-bot protections.-