clawifi-mcp
Clawifi MCP server provides web search, page/PDF reading, structured extraction, site mapping/crawling, credit estimates, async job tracking, and account/history/monitor access for AI agents.
Search the web with ranked results, snippets, providers, dates, and cache-hit handling.
Fetch one URL as Markdown, optionally returning only answer-relevant passages or the full page text, with browser rendering for JavaScript shells.
Scrape URLs with source-aware routing and optional passage selection.
Extract structured data from URLs using presets or a custom extraction schema.
Map discoverable URLs on a site, optionally ranked by search relevance.
Start asynchronous crawls and poll crawl status/results.
Estimate credit costs for billable operations without running or charging them.
Create and poll asynchronous jobs for billable operations.
View authenticated account profile/plan, credit usage/balance, request history, and configured Evidence Monitor checks.
Handles Cloudflare challenge pages and browser fingerprinting to enable web scraping and fetching through Cloudflare-protected sites.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@clawifi-mcp fetch the content of https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
clawifi
Official Python SDK, CLI, and MCP server for Clawifi, web search and page reading for AI agents: search many engines as one, read pages and PDFs, map and crawl sites, and get back only the passages that answer the question, each with its source. One typed API and MCP surface for all of it.
This package is a thin client over the hosted Clawifi API (hosted API version 1.12.1 at time of release); the Clawifi server is not included. Docs, pricing and status: clawifi.dev/docs · clawifi.dev/pricing · clawifi.dev/status · clawifi.dev/security.
One-click install (hosted server, no key needed to try)
Both add the hosted remote server https://api.clawifi.dev/mcp. No key to copy: the free tools work at once, and your client asks you to sign in (a free account, 1,000 credits a month) when a call needs one. One line for every other client (Claude Code, Codex, Claude, ChatGPT, Hermes): clawifi.dev/install.
Related MCP server: Apeiron
Agent skill
npx skills add ReARCLabs/clawifi-mcpInstalls skills/clawifi/SKILL.md into your coding agent: when to use Clawifi, which tool fits, and what each call costs. The same file is at clawifi.dev/agent-onboarding/skill.md.
Install
pip install clawifiGet an API key
clawifi signup --email you@example.com --password "a strong password"This creates an account, grants 1,000 free credits, and prints an API key. Export it:
export CLAWIFI_API_KEY="clawifi_live_..."Quicktest
clawifi quicktest --url https://example.comSet up your MCP client
clawifi initfinds the MCP clients installed on this machine (Claude Code, Codex, Cursor, Claude Desktop, VS Code) and prints, for each one, the install line for the local server (clawifi-mcp, stdio) and for the remote server, followed by a first call to try. It only prints; it never shows your key, only $CLAWIFI_API_KEY or a masked clawifi_live_****. clawifi init --all shows every client, found or not.
To have it edit a config file for you:
clawifi init --write cursor # ~/.cursor/mcp.json, stdio entry
clawifi init --write cursor --remote # ~/.cursor/mcp.json, remote entry
clawifi init --write claude-desktop # claude_desktop_config.json, stdio entry--write asks before it changes anything and copies the existing file to <file>.bak-<timestamp> first. Claude Code, Codex and VS Code have their own install commands, which init prints.
For Claude Code, for example:
# local (stdio)
claude mcp add --env CLAWIFI_API_KEY=$CLAWIFI_API_KEY --transport stdio clawifi -- clawifi-mcp
# remote
claude mcp add --transport http clawifi https://api.clawifi.dev/mcp --header "Authorization: Bearer $CLAWIFI_API_KEY"Use as an MCP server
The local MCP server runs as:
clawifi-mcpreading CLAWIFI_API_KEY from the environment. clawifi mcp config prints a minimal config for it. Every tool carries a title and the MCP hints readOnlyHint, destructiveHint, idempotentHint and openWorldHint; none is destructive.
The billable tools (fetch, scrape, extract, map_site, crawl, search_free) take their common options as parameters and any other API field in options, which is merged into the request body. They refuse dry_run: use the estimate tool to price a call. Arguments a tool does not declare are refused rather than ignored.
fetch takes content_scope ("main", the default, or "full" for the whole page text with scripts and styles removed) and render ("never", the default; "auto" opens the browser only for a JavaScript shell; "always" reads in the browser). A browser render costs 3 credits instead of 1, and estimate("fetch", {"url": ..., "render": "auto"}) quotes the range first. Any other value is refused and nothing is sent; both are sent only when they differ from the default. Each tool's description names the fields it returns (see Response fields).
Remote MCP
Clawifi also runs a hosted MCP server, so nothing needs installing:
URL:
https://api.clawifi.dev/mcp(streamable HTTP)Auth:
Authorization: Bearer <your key>, or theX-Clawifi-Api-Key: <your key>header
The old host api.clawifi.qzz.io (API and /mcp) keeps working, so CLAWIFI_API_BASE=https://api.clawifi.qzz.io and existing remote configs need no change.
Without a key the remote server offers a keyless tier of four tools, search_free, fetch_url, scrape_url and map_site, at 12 calls a minute and 50 a day per address. With a key you get the full tool list.
Credits
Calls are charged per operation (a plain fetch is 1 credit, a browser render 3, a crawl 1 per page, and so on). See clawifi.dev/pricing, or print the current table, which needs no key:
clawifi creditsThe credits MCP tool and client.credits() show your own usage and balance instead.
Use as a Python SDK
from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY from the environment
page = client.fetch("https://example.com")
print(page["success"])An async client is also available as clawifi.AsyncClawifi.
search_content searches and reads the top results; it is compact by default and full=True returns the full output:
results = client.search_content("agent memory benchmarks", full=True)Responses are returned as the API sends them, so fields such as published_at, modified_at and fetched_at come through untouched. Request fields are passed through as keyword arguments too:
page = client.fetch("https://example.com/pricing", content_scope="full") # the whole page text
page = client.fetch("https://app.example.com", render="auto") # browser only for a JavaScript shellResponse fields
Every response field, and the fields a call does not return, is listed on the endpoint pages: search, search with content, fetch, scrape and extract.
Fields worth knowing (the search cache since hosted API 1.11.0, the page fields since 1.12.0):
search_free:cache_hitistruewhen an identical search within 60 seconds was answered from the cache; such an answer costs 0 credits. Each result carriesurl,title,snippet,rank,providerand, when the engine gives a date,published_atandmodified_at.fetch:data.page_text_chars(how much readable text the page holds),data.content_chars(the length ofcontent_markdownbefore anyqueryselection),data.scope_used(main,whole,selectororfull) anddata.language(the page's<html lang>).mode_usedisbrowserwhenrender: "auto"rendered a JavaScript shell; with the defaultrender: "never"such a page answerscontent_status: "needs_javascript"and costs 0.
Estimates (hosted API 1.7.0 and later)
quote = client.estimate("fetch", url="https://example.com")
# {"dry_run": true, "route": "POST /fetch", "estimate": {"minimum": ..., "maximum": ..., "raises": [...]},
# "balance": ..., "would_402": ..., "would_429": ..., "binding_limit": ..., "charged": 0}
client.crawl("https://example.com", max_pages=50, dry_run=True)estimate() and dry_run=True work for fetch, scrape, extract, map, crawl, search_free and search_content. Before the first dry run the client checks the API version at /product/manifest; an API older than 1.7.0 would ignore dry_run and run (and bill) the call, so the client raises ClawifiDryRunUnsupportedError and sends nothing. Pre-releases such as 1.7.0rc1 count as older than 1.7.0.
dry_run is sent only when it is True, and only as the JSON boolean true: a normal call, or dry_run=False, has no dry_run key in its body, and a value that is not a bool raises TypeError before anything is sent. The API answers any dry_run key (even false) on a route that cannot dry-run with a 422, so schema_assist drops dry_run=False and refuses dry_run=True.
The estimate MCP tool does the same from an agent: estimate(operation="fetch", args={"url": "https://example.com"}).
Rate limits
A 429 raises ClawifiAPIError with retry_after (seconds, from the Retry-After header) and limit_scope (which limit was hit, from details.limit_scope). Retryable calls wait out a Retry-After of up to 10 seconds before retrying; a longer one is raised straight away.
MCP tools
Tool | What it does |
| Fetch one URL (page or PDF) as Markdown. With |
| Scrape a URL with source-aware routing and return structured content. With |
| Search the web and return ranked results ( |
| Extract structured data from a URL. |
| Map a site's discoverable URLs starting from a seed URL. |
| Start an asynchronous crawl from a seed URL; returns a |
| Price a |
| Get the current status and results of a crawl. |
| Create an asynchronous job for a billable operation; returns a |
| Get the current status and results of a job. |
| Get the authenticated account's profile and plan. |
| Get the authenticated account's credit usage and remaining balance. |
| List past requests for the authenticated account. |
| List configured Evidence Monitor checks for the authenticated account. |
Documentation
See clawifi.dev/docs for the full API reference, MCP tool list, and CLI command reference.
License
MIT
Available Tools
14 toolsaccountAccount profileARead-onlyIdempotentInspect
Get the authenticated account's profile and plan.
Inputs: no named parameters. Returns: object
crawlStart a crawlAInspect
Start an asynchronous crawl from a seed URL. Returns a crawl_id to poll. Each page is billed; use the estimate tool first to price it.
Inputs: seed_url (required), dry_run, options, max_depth, +4 more. Returns: object
create_jobCreate a jobAInspect
Create an asynchronous job for a billable operation. Returns a job_id to poll.
Inputs: payload (required), operation (required), dry_run. Returns: object
creditsCredit usageARead-onlyIdempotentInspect
Get the authenticated account's credit usage and remaining balance.
Inputs: no named parameters. Returns: object
estimateEstimate creditsARead-onlyIdempotentInspect
Price a billable call without running it or charging for it (Clawifi API 1.7.0+). operation is one of fetch, scrape, extract, map, crawl, search_free or search_content; args are that operation's fields, for example {"url": "https://example.com"}, {"seed_url": ...} for crawl or {"query": ...} for the searches. Returns estimate {minimum, maximum, raises}, balance, would_402, would_429 and binding_limit; charged is 0.
Inputs: operation (required), args. Returns: object
extractExtract structured dataBRead-onlyIdempotentInspect
Extract structured data from a URL, with an extraction schema or a preset (or neither, for the defaults).
Inputs: url (required), preset, dry_run, options, +2 more. Returns: object
fetchFetch a URLARead-onlyIdempotentInspect
Fetch one URL (page or PDF) as Markdown. query returns only the top_k passages that answer it. content_scope="full" returns the whole page text. render="auto" opens the browser only for a JavaScript shell (up to 3 credits). Returns data.title, content_markdown, content_text, page_text_chars, content_chars, scope_used, language and content_status; fields: https://clawifi.dev/docs/fetch
Inputs: url (required), query, top_k, render, +4 more. Returns: object
get_crawlGet crawl statusBRead-onlyIdempotentInspect
Get the current status and results of a crawl.
Inputs: crawl_id (required). Returns: object
get_jobGet job statusBRead-onlyIdempotentInspect
Get the current status and results of a job.
Inputs: job_id (required). Returns: object
list_historyList request historyARead-onlyIdempotentInspect
List past requests for the authenticated account, newest first. Filters: operation (e.g. search_free), status, kind and a created_after/created_before window (ISO 8601); pass next_cursor back as cursor.
Inputs: kind, limit, cursor, status, +3 more. Returns: object
list_monitorsList monitorsBRead-onlyIdempotentInspect
List configured Evidence Monitor checks for the authenticated account.
Inputs: limit, include_deleted. Returns: object
map_siteMap a site's URLsBRead-onlyIdempotentInspect
Map a site's discoverable URLs starting from a seed URL. search ranks them by relevance to a phrase.
Inputs: url (required), search, dry_run, options, +3 more. Returns: object
scrapeScrape a URLCRead-onlyIdempotentInspect
Scrape a URL with source-aware routing and return structured content. formats defaults to ["markdown"]. With query, markdown and text are only the top_k passages that answer it.
Inputs: url (required), mode, query, top_k, +6 more. Returns: object
search_freeFree web searchARead-onlyIdempotentInspect
Search the web and return ranked results: url, title, snippet, rank, provider, and published_at/modified_at when the engine gives a date. No page text; read a result with fetch. An identical search within 60 s returns cache_hit: true for 0 credits. Fields: https://clawifi.dev/docs/search
Inputs: query (required), dry_run, options. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 14 tools
Three tools (fetch, scrape, extract) all retrieve content from a single URL, and only fine wording distinguishes 'Markdown of a page' from 'structured content' from 'schema-based extraction'. There is also dual async machinery (crawl/get_crawl vs create_job/get_job) plus an `estimate` tool that mirrors every other operation, so boundaries are described but easy to misselect.
The set mixes consistent verb_noun names (get_crawl, create_job, get_job, map_site, list_history, list_monitors) with bare verbs (fetch, scrape, extract, crawl, estimate) and bare nouns (account, credits). It stays readable, but there is no single predictable pattern and polling is split between get_crawl and get_job.
14 tools is comfortably within a healthy range and each maps to a plausible capability of a scraping/search API (search, fetch, scrape, extract, map, crawl, pricing, jobs, account, credits, history, monitors). Nothing is bloated enough to be unwieldy.
Core read/lifecycle workflows (search, fetch, scrape, extract, map, crawl, async jobs, cost estimation, billing, history) are well covered. Minor gaps: no cancel/delete for jobs or crawls, and list_monitors has no create/get counterpart, leaving monitor management a dead end; estimate also references a search_content operation not exposed as a tool.
Maintenance
Related MCP Connectors
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
Web search API and MCP server for developers building AI agents.
Turn any public website into an MCP server for agents to search, read and navigate.
Related MCP Servers
- AlicenseAqualityBmaintenanceAn MCP server that enables AI assistants to fetch web content in multiple formats (HTML, JSON, text, Markdown) with intelligent content extraction, chunk management, and browser automation support.6165 npm15MIT
- AlicenseNot gradedqualityCmaintenanceUnlimited web access MCP server for AI agents, enabling search, fetch, and learning across multiple sources while bypassing anti-bot protections.MIT
- AlicenseNot gradedqualityCmaintenanceMCP tool server that gives any AI agent the ability to search, scrape, and analyze content across the internet.42MIT
- AlicenseNot gradedqualityDmaintenanceMCP server for web scraping and browser automation, enabling AI agents to extract clean, token-efficient content from web pages.1MIT