Octocrawl
Server Details
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
The last two health checks of this connector failed. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.
If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.
- Status
- Unhealthy
- Uptime
- 0.0% over the last 24 hours
- OAuth
- Works in Glama
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- Repository
- 77777R7/Octocrawl
- GitHub Stars
- 7
- Server Listing
- Octocrawl
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 17 tools
The four entry points (scrape, batch_scrape, crawl, map) are clearly differentiated, and descriptions explicitly cross-reference and negate each other ('Not for: many URLs or a whole site section' / 'Not for: reading pages'). Residual overlap exists between get_crawl's report and get_crawl_errors, and between get_crawl_pages and get_scrape, but the verb distinctions (report vs items vs errors vs single-call record) are spelled out.
Fifteen of seventeen tools follow a predictable snake_case verb_noun or noun_resource pattern (get_batch_items, wait_crawl, cancel_batch, list_active_crawls). The two deviations are 'batch_scrape' (reversed order versus scrape/crawl verbs) and the bare 'map', which carries neither a verb nor a clear noun-verb pairing.
Seventeen tools is slightly heavy but each async job type (crawl, batch) legitimately needs start/wait/status/items/errors/cancel, which accounts for most of the surface. A case could be made to fold get_crawl_errors into get_crawl or get_crawl_pages via a status filter.
The scrape→crawl→batch lifecycle is well covered with cancel and error paths for both async job types, but the surface is asymmetric: list_active_crawls has no batch counterpart, so an agent that loses a batch taskId cannot rediscover it. There is also no listing of recent scrapes beyond the 10-minute get_scrape lookup.
Available Tools
17 toolsbatch_scrapeStart a batch scrapeAInspect
Run 1-1000 known URLs as one persistent job. Returns { taskId } at once; wait_batch waits for it, get_batch_items pages the results and get_batch_errors lists the failures and robots refusals. Each URL takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.
Inputs: urls (required), mode, retry, mobile, +18 more. Returns: taskId, replayed
cancel_batchCancel a batchBDestructiveIdempotentInspect
Cancel a batch by its taskId; the items already read stay readable.
Inputs: id (required).
cancel_crawlCancel a crawlBDestructiveIdempotentInspect
Cancel a crawl task. Completed pages remain queryable.
Inputs: id (required).
crawlStart a crawlAInspect
Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery. On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.
Inputs: url (required), mode, mobile, verify, +28 more. Returns: taskId, replayed
get_batchGet batch statusARead-onlyIdempotentInspect
Read a batch's status by its taskId: its state, item counts and the items waiting for the person (waitingForPerson).
Inputs: id (required). Returns: status, taskId
get_batch_errorsGet batch errorsARead-onlyIdempotentInspect
List a batch's items that did not succeed (status failed, blocked, cancelled or budget_exceeded), each as get_batch_items lists it: { items, nextCursor, hasMore }, in pages of up to 100 (cursor, limit). A URL robots.txt refused is among them, blocked with its blockReason. notServed lists pages read but not the one asked for.
Inputs: id (required), debug, limit, cursor.
get_batch_itemsGet batch itemsARead-onlyIdempotentInspect
Page through a batch's results by its taskId (cursor, limit up to 50). fields keeps only the named item fields (plus id, url, status, createdAt), e.g. ["json"] for a JSON batch. With maxResults the tool follows the cursors itself and answers { items, nextCursor, hasMore, stoppedBy }.
Inputs: id (required), debug, limit, cursor, +2 more. Returns: items, hasMore, stoppedBy, nextCursor
get_crawlGet crawl statusARead-onlyIdempotentInspect
Read a crawl's report by its taskId: its status, page counts and the discovery counters of the links it collapsed or refused.
Inputs: id (required). Returns: status, taskId
get_crawl_errorsGet crawl errorsCRead-onlyIdempotentInspect
Read a paginated list of crawl errors by task id.
Inputs: id (required), debug, limit, cursor, +2 more.
get_crawl_pagesGet crawl pagesARead-onlyIdempotentInspect
Page through a crawl's pages by task id (the latest attempt's unless attemptId is given). fields keeps only the named page fields (plus id, url, status, createdAt); debug adds the routing audit and trace; includeDuplicates lists the pages whose content repeated an earlier page's. With maxResults the tool follows the cursors itself and answers { items, nextCursor, hasMore, stoppedBy }.
Inputs: id (required), debug, limit, cursor, +4 more. Returns: items, hasMore, stoppedBy, nextCursor
get_scrapeGet a scrape recordARead-onlyIdempotentInspect
Read one scrape call's record by its scrapeId (metadata.scrapeId on the response): the request, the verdict, metadata, warnings and hints; no page body. Kept 10 minutes.
Inputs: id (required). Returns: status, scrapeId, blockReason, failureReason
list_active_crawlsList active crawlsARead-onlyIdempotentInspect
List this account's crawls that are queued or running, oldest first: each with its id, start URL, status and pages so far. Empty when none is.
Inputs: no named parameters.
mapMap a siteARead-onlyIdempotentInspect
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.
Inputs: url (required), mode, debug, limit, +11 more. Returns: id, links, counts, status, +3 more
scrapeScrape a pageARead-onlyInspect
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step. metadata.scrapeId names the call's record for get_scrape. One scrape at a time in the browser per key or sign-in: a parallel one that needs it gets browser_busy, so send many URLs as one batch_scrape.
Inputs: url (required), mode, debug, mobile, +17 more. Returns: status, finalUrl, markdown, readiness, +8 more
scrape_productAmazon.sg product JSONBRead-onlyIdempotentInspect
Get evidence-backed JSON for one anonymous Amazon.sg /dp/{ASIN} product. No schema or model setup needed.
Inputs: url (required), debug.
wait_batchWait for a batchARead-onlyIdempotentInspect
Wait up to timeoutMs (default 30000, at most 55000) for a batch to finish and return its status; at the timeout it returns the status so far. Its rate (pagesPerMinute, etaSeconds) says how long the rest should take, so wait again or read get_batch_items.
Inputs: id (required), timeoutMs. Returns: status, taskId
wait_crawlWait for a crawlARead-onlyIdempotentInspect
Wait up to timeoutMs (default 30000, at most 55000) for a crawl to finish and return its report; at the timeout it returns the report so far, so wait again or read get_crawl_pages.
Inputs: id (required), timeoutMs. Returns: status, taskId
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Web pages as Markdown, text or HTML, plus Google Maps places and reviews, for AI agents.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Clean web data for agents: any URL to Markdown or JSON, plus search, crawls and company brand data.
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch and crawl web pages, extract readable content, and convert it into clean Markdown with preserved links while minimizing token usage.MIT
- AlicenseNot gradedqualityAmaintenanceFetch web pages into clean, RAG-ready Markdown with a verifiable provenance record — source URL, UTC fetch time, SHA-256 of the response body, redirect chain, and an extraction-fidelity metric. Writes WARC archives and honours robots.txt/ai.txt, so every document an agent reads can be re-verified and cited.8 PyPI2MIT
- AlicenseNot gradedqualityCmaintenanceEnables converting any URL or website into clean, LLM-ready Markdown, supporting single-page scraping and full-site crawling for AI agents.MIT
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch webpages or crawl sites, extract main content, and convert it into clean Markdown with links preserved while minimizing token use.MIT
Glama MCP Gateway
Add one secure layer between your agents and this server.