Skip to main content
Glama

Server Details

Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.

Ownership verified

The last two health checks of this connector failed. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.

If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.

Status
Unhealthy
Uptime
0.0% over the last 24 hours
OAuth
Works in Glama
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
Repository
77777R7/Octocrawl
GitHub Stars
7
Server Listing
Octocrawl
ChatGPT
–
Claude
–

TDQS

A3.6/5.0

Scored across 17 tools

Disambiguation4/5

The four entry points (scrape, batch_scrape, crawl, map) are clearly differentiated, and descriptions explicitly cross-reference and negate each other ('Not for: many URLs or a whole site section' / 'Not for: reading pages'). Residual overlap exists between get_crawl's report and get_crawl_errors, and between get_crawl_pages and get_scrape, but the verb distinctions (report vs items vs errors vs single-call record) are spelled out.

Naming Consistency4/5

Fifteen of seventeen tools follow a predictable snake_case verb_noun or noun_resource pattern (get_batch_items, wait_crawl, cancel_batch, list_active_crawls). The two deviations are 'batch_scrape' (reversed order versus scrape/crawl verbs) and the bare 'map', which carries neither a verb nor a clear noun-verb pairing.

Tool Count4/5

Seventeen tools is slightly heavy but each async job type (crawl, batch) legitimately needs start/wait/status/items/errors/cancel, which accounts for most of the surface. A case could be made to fold get_crawl_errors into get_crawl or get_crawl_pages via a status filter.

Completeness3/5

The scrape→crawl→batch lifecycle is well covered with cancel and error paths for both async job types, but the surface is asymmetric: list_active_crawls has no batch counterpart, so an agent that loses a batch taskId cannot rediscover it. There is also no listing of recent scrapes beyond the 10-minute get_scrape lookup.

Available Tools

17 tools
batch_scrapeStart a batch scrapeAInspect

Run 1-1000 known URLs as one persistent job. Returns { taskId } at once; wait_batch waits for it, get_batch_items pages the results and get_batch_errors lists the failures and robots refusals. Each URL takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.

Inputs: urls (required), mode, retry, mobile, +18 more. Returns: taskId, replayed

cancel_batchCancel a batchB
DestructiveIdempotent
Inspect

Cancel a batch by its taskId; the items already read stay readable.

Inputs: id (required).

cancel_crawlCancel a crawlB
DestructiveIdempotent
Inspect

Cancel a crawl task. Completed pages remain queryable.

Inputs: id (required).

crawlStart a crawlAInspect

Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery. On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.

Inputs: url (required), mode, mobile, verify, +28 more. Returns: taskId, replayed

get_batchGet batch statusA
Read-onlyIdempotent
Inspect

Read a batch's status by its taskId: its state, item counts and the items waiting for the person (waitingForPerson).

Inputs: id (required). Returns: status, taskId

get_batch_errorsGet batch errorsA
Read-onlyIdempotent
Inspect

List a batch's items that did not succeed (status failed, blocked, cancelled or budget_exceeded), each as get_batch_items lists it: { items, nextCursor, hasMore }, in pages of up to 100 (cursor, limit). A URL robots.txt refused is among them, blocked with its blockReason. notServed lists pages read but not the one asked for.

Inputs: id (required), debug, limit, cursor.

get_batch_itemsGet batch itemsA
Read-onlyIdempotent
Inspect

Page through a batch's results by its taskId (cursor, limit up to 50). fields keeps only the named item fields (plus id, url, status, createdAt), e.g. ["json"] for a JSON batch. With maxResults the tool follows the cursors itself and answers { items, nextCursor, hasMore, stoppedBy }.

Inputs: id (required), debug, limit, cursor, +2 more. Returns: items, hasMore, stoppedBy, nextCursor

get_crawlGet crawl statusA
Read-onlyIdempotent
Inspect

Read a crawl's report by its taskId: its status, page counts and the discovery counters of the links it collapsed or refused.

Inputs: id (required). Returns: status, taskId

get_crawl_errorsGet crawl errorsC
Read-onlyIdempotent
Inspect

Read a paginated list of crawl errors by task id.

Inputs: id (required), debug, limit, cursor, +2 more.

get_crawl_pagesGet crawl pagesA
Read-onlyIdempotent
Inspect

Page through a crawl's pages by task id (the latest attempt's unless attemptId is given). fields keeps only the named page fields (plus id, url, status, createdAt); debug adds the routing audit and trace; includeDuplicates lists the pages whose content repeated an earlier page's. With maxResults the tool follows the cursors itself and answers { items, nextCursor, hasMore, stoppedBy }.

Inputs: id (required), debug, limit, cursor, +4 more. Returns: items, hasMore, stoppedBy, nextCursor

get_scrapeGet a scrape recordA
Read-onlyIdempotent
Inspect

Read one scrape call's record by its scrapeId (metadata.scrapeId on the response): the request, the verdict, metadata, warnings and hints; no page body. Kept 10 minutes.

Inputs: id (required). Returns: status, scrapeId, blockReason, failureReason

list_active_crawlsList active crawlsA
Read-onlyIdempotent
Inspect

List this account's crawls that are queued or running, oldest first: each with its id, start URL, status and pages so far. Empty when none is.

Inputs: no named parameters.

mapMap a siteA
Read-onlyIdempotent
Inspect

List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.

Inputs: url (required), mode, debug, limit, +11 more. Returns: id, links, counts, status, +3 more

scrapeScrape a pageA
Read-only
Inspect

Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step. metadata.scrapeId names the call's record for get_scrape. One scrape at a time in the browser per key or sign-in: a parallel one that needs it gets browser_busy, so send many URLs as one batch_scrape.

Inputs: url (required), mode, debug, mobile, +17 more. Returns: status, finalUrl, markdown, readiness, +8 more

scrape_productAmazon.sg product JSONB
Read-onlyIdempotent
Inspect

Get evidence-backed JSON for one anonymous Amazon.sg /dp/{ASIN} product. No schema or model setup needed.

Inputs: url (required), debug.

wait_batchWait for a batchA
Read-onlyIdempotent
Inspect

Wait up to timeoutMs (default 30000, at most 55000) for a batch to finish and return its status; at the timeout it returns the status so far. Its rate (pagesPerMinute, etaSeconds) says how long the rest should take, so wait again or read get_batch_items.

Inputs: id (required), timeoutMs. Returns: status, taskId

wait_crawlWait for a crawlA
Read-onlyIdempotent
Inspect

Wait up to timeoutMs (default 30000, at most 55000) for a crawl to finish and return its report; at the timeout it returns the report so far, so wait again or read get_crawl_pages.

Inputs: id (required), timeoutMs. Returns: status, taskId

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and LLM pipelines to fetch and crawl web pages, extract readable content, and convert it into clean Markdown with preserved links while minimizing token usage.
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Fetch web pages into clean, RAG-ready Markdown with a verifiable provenance record — source URL, UTC fetch time, SHA-256 of the response body, redirect chain, and an extraction-fidelity metric. Writes WARC archives and honours robots.txt/ai.txt, so every document an agent reads can be re-verified and cited.
      8 PyPI
      2
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and LLM pipelines to fetch webpages or crawl sites, extract main content, and convert it into clean Markdown with links preserved while minimizing token use.
      MIT
    Try in Browser

    Glama MCP Gateway

    Add one secure layer between your agents and this server.