Spicrawl

Append URLs to a batch job that was submitted with open: true. Same item vocabulary as spicrawl_batch_submit (EITHER urls or items, not both, plus shared settings for these new items). Fails with CONFLICT if the job is closed or finished. Not idempotent: calling twice adds the URLs twice. Returns { job, items_added, items_dispatched }.

Cancel a batch job: items not yet started are cancelled and will never run; items already finished keep their results; items already running finish first, so the job comes back cancelling and moves to cancelled when they drain. Cannot be undone — resubmit to run the URLs again. Returns the job object.

Stop an open batch job accepting items, so it completes once its queued work drains. Idempotent (closing a closed job succeeds). Returns the job object.

List this API key's project's batch jobs, newest first, with status and progress. Returns { batches: [...], next_cursor? }; pass next_cursor back as cursor for the next page.

Return a page of FINISHED items of a batch job (one record per item: seq, url, external_id, status, error, and content, or a result_url API path (GET /v1/batch/{id}/tasks/{seq}/content, same key) for large bodies). Unfinished items are not listed yet. Returns { results: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Results expire after the job's retention window (410 Gone).

Re-queue every FAILED item of a batch job so it runs again (succeeded items are untouched; a job with no failures is returned unchanged). Returns { job, items_reset, items_dispatched }.

Return one batch job with its status and progress (total/completed/succeeded/failed/remaining, credits charged). Poll this after spicrawl_batch_submit; once status is completed, failed or cancelled, read spicrawl_batch_results.

Submit many URLs to be scraped as one asynchronous job, with shared settings applied to every URL. Give EITHER urls or items, not both (at least one URL, unless open is true). Returns the job object (id, status, estimated_credits, progress). Poll with spicrawl_batch_status, read with spicrawl_batch_results or spicrawl_batch_task_content. Use this instead of many spicrawl_scrape calls when you have tens or thousands of URLs. Set open: true to keep the job accepting more URLs via spicrawl_batch_add_items; an open job never finishes until you call spicrawl_batch_close.

Return the full payload one item of a batch job produced, by its seq (items are numbered from 0 in submission order). Cheaper than paging results when you need one URL's output. A CONFLICT error means either the item has not finished yet (poll spicrawl_batch_status and ask again) or it finished as failed (it will never produce content; the message says which). NOT_FOUND means no such seq.

Mints a single-use connect URL for Spicrawl's remote browser (GET /v1/browser, a DevTools Protocol (CDP) WebSocket) so YOUR code can drive it with any CDP client, using the URL as the browser WebSocket endpoint. It calls POST /v1/browser/token and returns a ws:// or wss:// URL carrying a short-lived token, never the API key: the URL works ONCE, within 60 seconds, and only with the options given here (editing them in the URL is refused), so call this right before connecting and again for every new session. Prefer sending the key as an Authorization header when the client supports headers on the WebSocket upgrade: then connect to the plain /v1/browser URL with 'Authorization: Bearer ' and no token is needed. The key must carry the browser scope. Parameter errors are reported when minting. The result's session_id names the session once connected: when a page needs a person (a login, a one-time code, a captcha), pass it to spicrawl_browser_live_view for a link they can open, and to spicrawl_browser_control to see when they hand control back.

Reads or changes who drives a connected remote browser: agent (your CDP connection) or human (a person in its live view, see spicrawl_browser_live_view). status reads it: poll it after sending a live-view link, and continue once controller is agent again. takeover hands control to the person before they ask (your input and navigation then fail with error -32001); release takes it back. The person can also take and give back control from the live view themselves.

Gets a link a person can open to watch the remote browser your CDP connection is driving, and take control of it: the same tabs, cookies and page. Use it when a page needs a human: a login, a one-time code, a captcha. Send them live_view_url; they press "Take control", do the step, and give control back. While they have it, your connection's input and navigation commands fail with error -32001 (reading the page still works); poll spicrawl_browser_control (action status) until controller is agent, then carry on. The link works once and must be opened within 60 seconds; ask again for a new one. The session must still be connected. Watching costs nothing beyond the session itself.

The docs' llms.txt: every Spicrawl documentation page with its title, a one-line description and its Markdown URL. Use it to find the right page when a search does not, or to get an overview of what is documented; read a page with spicrawl_docs_read. No parameters.

Fetches one Spicrawl documentation page as Markdown. path may be a page path (guides/anti-bot), a docs URL from spicrawl_docs_search (url or md_url), or either with a #anchor (the whole page is returned; the anchor names the section to look at). Only pages on the Spicrawl docs (https://docs.spicrawl.com/docs) can be read. Pages over 60000 characters are truncated, and the text says so. Don't know the path? Use spicrawl_docs_search, or spicrawl_docs_index (llms.txt) for the list of every page.

Full-text search over the Spicrawl documentation (guides, API reference, errors, limits). Use it to look up a parameter name, value or combination for a spicrawl_* tool, or to explain an error a spicrawl_* tool returned: search the error code itself (e.g. ERR::UPSTREAM::CHALLENGE) and the result also carries a direct link to that code's entry on the errors page. Returns the best-matching pages, each with its title, the matching sections (heading, a short snippet, URL) and md_url; then call spicrawl_docs_read with a page's md_url or url to read it in full.

The full log record of one past API call by its request id. Every response carries its id in the X-Request-Id header (and error bodies carry it too), so this is the tool for debugging a failed or slow scrape: it shows status, error_code/error_detail, attempts, the engine and proxy (tier, country, provider, sticky) actually used, whether and how the target blocked it (vendor/signal/rule), per-stage latency spans and credits charged. Returns not_found both for unknown ids and for ones older than your retention window, and for another project's request unless your key has the read scope.

Recent API calls in your key's project, newest first, each with project_id, status, engine, http_status, error_code/error_detail, attempts, URL, proxy used, latency spans (queue/engine/upstream/total ms), block detection and credits. Use it to find failing calls (only_errors=true) or to spot patterns across a target. all_projects=true lists every project in your organization instead (needs a key with the read scope; 403 ERR::AUTH::INSUFFICIENT_SCOPE otherwise). Records only exist within the retention window reported in page.retention_hours / page.retained_since; an empty page means nothing in that window. Paginate by passing page.next_before and page.next_before_id back as before and before_id while page.has_more is true, keeping all_projects the same. Not billing-grade: use spicrawl_usage for totals.

Retrieve a web page and return its content as markdown (default), text, HTML, a JSON envelope, or a printed PDF. Handles the fetch vs. browser decision, charset decoding, and PDF-to-text for you. Optionally extract structured data with CSS selectors (extract) or autoparse (a natural-language ai_extract is coming soon), return discovered links, take a screenshot, or scope the DOM with include_tags/exclude_tags. Each successful call is billed in credits (a failure costs 0). Results are cached by default, and a cache hit is billed like the fetch that stored it (the cache saves time, not credits); set cache=false for time-sensitive pages. With the defaults it only reads the page. A method other than GET, HEAD or OPTIONS, and the actions that click, fill or run scripts on the page, can change state on the target site: set them only when the user asks for that. For markdown, text and HTML it returns { content }. With format: "json", or when extract, ai_extract, autoparse, links, screenshot or network_capture is set, it returns the JSON envelope: content, the site's status, credits, engine (the rendering tier: fetch, js or browser), warnings and data for extraction. Screenshots come back as image content blocks; each screenshots entry in the JSON keeps its metadata and says which block holds it. A PDF comes back as a resource content block, with its size, the rendering tier and the credits in the JSON.

Return the full stored state of a LIVE session — cookies, localStorage and other credentials — plus its fingerprint and domain scores. The session_context object can be passed to spicrawl_session_create to clone the session. The result contains secrets (login cookies): do not echo it to the user or logs unless asked.

Create a session: a persistent browser identity (cookie jar, localStorage, a fixed rendering tier and, by default, a sticky exit IP) that survives across scrapes. Use it for multi-step flows — log in, then fetch pages behind the login — by passing the returned id as session_id to every spicrawl_scrape call that should share the state. The rendering tier is fixed for the session's lifetime and every scrape on the session runs on it. Leave engine out and one is chosen for you. Release the session with spicrawl_session_release when done. Returns the session object (id, engine, status, sticky_key, proxy, expires_at).

Permanently delete a session and its stored state, including the record itself (prefer spicrawl_session_release if you only want to stop using it). Idempotent: deleting an already-deleted session succeeds. Refused while a scrape holds the session's lease unless force is true.

Return one session's metadata: status, engine (its rendering tier), proxy/exit, usage count, expiry, lease holder, and a summary of its stored context. Never returns the credentials themselves (use spicrawl_session_context for that).

List this API key's project's sessions, newest first. Returns { sessions: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Never includes the stored credentials.

End a session when you are done with it: frees its exit IP and browser state; it can no longer be used with spicrawl_scrape. Its stored cookies and storage are purged and cannot be recovered; start a new session and log in again to continue. The record stays visible (status released) so its history can still be inspected. Refused while a scrape holds the session's lease unless force is true. Returns the session object.

Billing-grade usage for your org over a date window, from the daily rollup (usage_daily), grouped by day, project, engine, feature or key. group_by=key returns per API key (key_id, name, prefix, project_id) counters per metric per day, from usage_key_daily; keyless (dashboard) usage is not in it, and a project-scoped key sees only its own project's keys. Use it to answer 'how many requests/credits did I use, and where'. The window defaults to the last 30 days including today; to is EXCLUSIVE (a single day 2026-08-01 is from=2026-08-01, to=2026-08-02) and the window may not exceed 400 days. The response carries stale_seconds (age of the newest rollup row; null when the window is empty). Requires the key to have the read scope.

Drift report for ONE day: re-derives usage from that day's request log and compares it with the billed daily rollup, so you can see whether the rollup missed or double-counted anything. Defaults to yesterday (UTC); today is still accumulating and would always show drift. Requires the read scope.

A compact current-period summary of your org's usage (totals as of now, from the daily rollup). No parameters. Use for a quick 'how much have I used so far'; use spicrawl_usage for a windowed breakdown. Requires the read scope.